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ABSTRACT 


Medical  leadership  in  the  military  health  system  (MHS)  has  been  identified  as  an  area  of 
concern.  Development  and  assessment  of  medical  leadership  among  uniformed  providers 
continues  to  pose  unique  challenges  to  units  and  organizations  within  the  MHS.  The  Uniformed 
Services  University  (USU)  has  been  identified  and  tasked  by  senior  Department  of  Defense 
(DoD)  leaders  to  bring  renewed  emphasis  on  developing  leadership  especially  among  student 
physicians.  Currently,  the  Department  of  Military  and  Emergency  Medicine  (MEM),  the 
academic  department  responsible  for  leadership  development  and  assessment,  has  no  leadership 
development  instrument  designed  to  assess  medical  leadership  development  in  student 
physicians  during  the  pre -residency  years. 

This  doctoral  project  created  a  medical  leadership  assessment  instrument  through  a 
standard  Delphi  Method.  A  panel  of  civilian  and  military  leaders  served  as  experts  who  provided 
input  on  the  development  and  critique  of  a  brief  leadership  assessment  instrument  through  a 
series  of  online  surveys.  After  the  draft  assessment  was  finalized  through  expert  consensus,  an 
additional  panel  of  potential  users  at  USU  participated  in  “cognitive  interviews,”  where  the 
instrument  was  reviewed  and  critiqued  for  accessibility,  style,  and  clarity.  The  new  instrument, 
called  the  Military  Medical  Leadership  Assessment  (MMLA),  is  discussed  in  relation  to  existing 
leadership  theories  commonly  used  within  the  military  and  at  USU.  Finally,  the  future  of  the 
MMLA  as  a  valuable  tool  to  improve  civilian  and  military  clinical  practice  is  also  discussed. 
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CHAPTER  1:  Introduction  &  Background 

Introduction 

In  2007,  The  Washington  Post  published  a  series  of  controversial  articles  that  outlined  a 
number  of  problems  with  patient  care  at  Walter  Reed  Army  Medical  Center  (WRAMC).  Within 
the  expose.  The  Post  outlined  several  problems  at  one  of  the  Anny’s  most  visible  treatment 
facility.  With  the  wars  in  Iraq  and  Afghanistan  reaching  a  fever  pitch,  several  hundred  wounded 
warriors  and  their  families  were  discovered  to  be  living  and  being  treated  in  conditions  that  were 
plagued  with  unsanitary  conditions.  In  a  piece  that  would  later  earn  The  Post  a  Pulitzer  Prize, 
the  articles  described  conditions  that  included  rodent  infested  buildings,  poor  security, 
incompetent  staff  (92;  136),  and  evidence  of  a  cover-up  once  the  story  gained  traction  in  the 
media  (100).  The  alleged  failures  at  the  Army’s  premier  healthcare  facility  would  ultimately 
lead  then-Secretary  of  Defense  Robert  M.  Gates  to  remove  the  Commander  of  WRAMC,  the 
Army  Surgeon  General,  and  the  Secretary  of  the  Army.  In  a  brief  with  reporters,  Secretary  Gates 
simply  stated,  “The  problems  at  Walter  Reed  appear  to  be  problems  of  leadership''  (emphasis 
added)  (156). 

In  response,  the  Army  selected  BG  Eric  Schoomaker,  M.D.,  Ph.D.,  then-commander  of 
Eisenhower  Army  Medical  Center,  Ft,  Gordon,  GA  to  take  command  of  WRAMC  and  take 
immediate  corrective  action.  In  recounting  the  Walter  Reed  scandal  and  the  Army’s  response, 
LTG  (R)  Schoomaker,  who  would  later  be  appointed  the  Army  Surgeon  General,  stated  that 
many  of  the  lessons  about  patient  care  and  healthcare  leadership  learned  in  Korea,  Vietnam,  and 
Operation  Desert  Storm  were  forgotten  or  ignored,  especially  in  terms  of  medical  evacuation 
(MEDEVAC),  patient  administration,  and  caring  for  the  families  of  wounded  Warriors  (160).  In 
an  interview  with  the  author  of  this  dissertation,  LTG  (R)  Schoomaker  stated  that  during  his 
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assessment  of  the  faculty  and  staff  of  WRAMC  several  key  problems  emerged.  In  the  same 
interview,  CSM  (R)  Althea  Green-Dixon  and  COL  (R)  Charles  Callahan  (who  were  LTG  (R) 
Schoomaker’s  senior  enlisted  adviser  and  chief  of  staff,  respectively)  confirmed  that  among  the 
most  glaring  problems  at  WRAMC  were  a  lack  of  communication,  lack  of  mission  focus,  and 
lack  of  leadership  assessment.  All  three  leaders  reported  that  among  the  first  courses  of  action 
performed  to  address  the  shortcomings  at  WRAMC  were  to  personally  assess  the  leadership 
ability  of  each  of  the  leaders  at  Walter  Reed  beyond  simple  medical  skills  and  clinical 
competence.  Additionally,  LTG  (R)  Schoomaker  brought  in  leadership  and  communications 
experts  as  outside  consultants  to  advise  the  new  leadership  team  how  to  best  identify  and  assign 
leaders  of  high  integrity,  responsibility,  and  motivation  to  greatly  improve  patient  administration, 
medical  staff  accountability,  and  resources  for  patient  families.  The  changes  to  WRAMC ’s 
leadership  culture  were  essential  to  enabling  the  organization  to  quickly  recover,  improve  patient 
care,  and  ultimately  conduct  a  successful  merge  with  the  National  Naval  Medical  Center  to 
create  the  Walter  Reed  National  Military  Medical  Center  (WRNMMC).  Although  the  Walter 
Reed  scandal  was  overshadowed  by  the  Iraqi  surge  and  the  2008  presidential  elections,  the 
military  medical  community  would  soon  be  rocked  by  another  tragedy. 

November  5th,  2009  was  a  beautiful  fall  day  at  Ft.  Hood,  Texas  where  nearly  50,000 
troops  lived,  trained,  and  continuously  deployed  to  Iraq  and  Afghanistan.  The  otherwise  placid 
midmorning  was  shattered  by  the  sounds  of  semi-automatic  gunfire.  Ordinarily,  sounds  of 
gunfire  and  explosions  would  have  been  considered  fairly  commonplace  at  the  sprawling  Army 
base,  but  these  bullets  were  not  fired  at  the  hundreds  of  ranges  that  are  scattered  throughout  the 
instillation.  The  person  shooting,  although  a  Soldier  assigned  to  Ft.  Hood,  was  not  aiming  at 
paper  targets  but  rather  at  his  fellow  Soldiers  who  were  processing  for  eventual  deployment  to 
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Afghanistan.  The  shooter,  MAJ  Nidal  Malik  Hasan,  had  that  morning  given  his  possessions 
away,  donned  the  white  robes  of  an  Islamic  martyr,  and  then  proceeded  to  perpetuate  what  then- 
Senator  Joe  Lieberman  (I-CT)  described  as  “the  worst  terrorist  attack  since  9/11”  (112). 

Although  the  attack  lasted  only  10  minutes  before  Hasan  was  critically  felled  by  military  police 
returning  fire,  13  of  his  fellow  Soldiers  lay  dead  with  30  others  seriously  wounded. 

With  Hasan  in  custody,  the  Army,  the  Department  of  Defense  (DoD),  and  the  Federal 
Bureau  of  Investigation  (FBI)  began  an  extensive  investigation  into  the  background  and 
motivation  behind  the  worst  act  of  violence  on  a  military  instillation  in  234  years.  One  of  the 
most  unnerving  facts  about  the  2009  Ft.  Hood  shooting  is  that  MAJ  Nidal  Malik  Hasan  was  a 
board  certified  psychiatrist  and  a  graduate  of  the  Uniformed  Services  University  (USU)  School 
of  Medicine  (SoM),  USU’s  Masters  of  Public  Health  program,  and  had  recently  completed  a 
fellowship  with  the  Center  for  Traumatic  Stress  at  USU.  When  coupled  with  the  fact  that  Hasan 
had  enlisted  in  1988  after  graduating  high  school  in  suburban  Virginia,  Hasan  had  served  over  20 
years  in  uniform.  Numerous  political  pundits,  politicians,  and  the  families  of  the  wounded  and 
murdered  Soldiers  pointedly  asked  the  Army  if  there  had  been  warning  signs  of  Hasan’s 
radicalization  over  two  decades  of  service  and  what  could  be  done  to  prevent  such  attacks  in  the 
future.  Investigative  journalists  from  the  Los  Angeles  Times ,  CBS,  ABC,  and  The  New  York 
Times  began  to  paint  a  picture  of  a  deeply  troubled  individual,  socially  isolated,  and  increasingly 
vocal  about  his  support  for  Islamic  extremism  (2;  124;  135).  Former  defense  and  intelligence 
officials  criticized  the  Army  Medical  Department  (AMEDD)  for  not  acting  on  warning  signs  of 
Hasan’s  increasing  radicalization,  workplace  outbursts,  praise  for  terror,  and  overall  mediocre 
performance  as  a  physician  (112;  151).  Then-Defense  Secretary  Robert  Gates  directed  an 
extensive  report  in  2010  to  address  how  the  Army  handled  Hasan  throughout  his  career  and  any 
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shortcomings.  Chaired  by  former  Secretary  of  the  Army  Togo  West,  the  report  (60)  directly 
criticized  the  AMEDD  and  USU  stating  “A  related  issue  involves  the  alleged  perpetrator ’s 
documented  performance  in  official  records  and  his  actual  performance  during  his  training, 
residency,  and  fellowship.  Some  signs  were  clearly  missed;  others  ignored.  ”  The  reports 
recommended  courses  of  action  for  the  AMEDD  and  USU  were  similarly  pointed:  “We  believe 
that  some  medical  officers  failed  to  apply  appropriate  judgment  and  standards  of  officer  ship 
with  the  alleged  perpetrator.  These  individuals  failed  to  demonstrate  that  officer  ship  is  the 
essence  of  being  a  member  of  the  military  profession,  regardless  of  the  officer ’s  specialty.  We 
also  found  that  some  medical  officers  failed  to  include  the  alleged  perpetrator’s  overall 
performance  as  an  officer. ..in  his  formal  performance  evaluations.  An  individual’s  total 
performance,  academic  and  non-academic,  in  a  school  environment  must  be  a  part  of  the 
formal  performance  evaluation  process  to  preclude  decisions  on  that  individual ’s  career  from 
being  flawed  because  of  incomplete  information.  ’’  (Emphasis  added). 

In  the  aftermath  of  the  Water  Reed  scandal  and  the  Ft.  Hood  shooting,  the  DoD  began  to 
re-emphasize  the  importance  of  military  medical  leadership  and  require  a  higher  level  of 
accountability  among  military  healthcare  leaders.  In  early  2009,  the  DoD  directed  the  Military 
Health  System  (MHS)  Office  of  Transformation  to  partner  with  the  Rand  Corporation  to  identify 
problems  in  developing  military  healthcare  leadership  and  make  appropriate  recommendations. 
In  201 1,  the  Rand  Corporation  and  the  MHS  Office  of  Transformation  (now  known  as  the 
Defense  Health  Agency  [DHA])  released  the  results  of  a  three  year  study  that  outlined  a  number 
of  problems  in  military  healthcare  leadership  development.  The  project,  Developing  Custodians 
of  Care,  surveyed  nearly  100  leaders  in  military  and  civilian  healthcare  settings.  Among  the 
challenges  identified  by  healthcare  leaders  were  the  inflexible,  uneven,  and  outdated  process  by 
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which  healthcare  leaders  are  developed  and  the  lack  of  any  evaluation  instrument  to 
accurately  assess  healthcare  leadership  (103). 

The  actions  surrounding  the  shortcomings  of  Walter  Reed,  the  Ft  Hood  Shooting  report, 
and  The  Rand  Corporation  study  have  a  common  thread  that  emphasizes  the  importance  of 
leadership  in  military  medicine  and  unambiguously  asserts  that  healthcare  leadership  is  critical  to 
the  future  success  of  uniformed  healthcare.  Additionally,  these  same  sources  point  to  healthcare 
leadership  assessment  as  an  essential  component  of  ensuring  continued  success  in  the  future  of 
military  healthcare.  However,  before  such  a  leadership  evaluation  instrument  is  created,  the 
current  consensus  of  experienced  leaders  must  be  collected  to  identify  the  following: 

1)  What  do  expert  and  successful  leaders  believe  is  important  in  assessing  leadership? 

2)  How  would  a  potential  leadership  assessment  instrument  be  created  from  expert 
consensus  and  guide  leadership  education  and  development  in  military  medicine? 

3)  Can  the  expert  consensus  be  incorporated  into  medical  education  and  leadership 
development? 

The  present  research  project  aimed  to  address  these  important  questions  by  engaging 
multiple  levels  of  leadership  within  the  Military  Health  System  (MHS)  and  civilian  community 
to  create  a  new  leadership  assessment  instrument.  It  is  important  to  note  that  this  doctoral 
project  was  developed  as  part  of  a  much  larger  effort  at  the  Uniformed  Services  University 
(USU).  As  a  result  of  the  leadership  challenges  and  needs  identified  in  the  past  decade  by  the 
Department  of  Defense  and  MHS,  USU  has  reenergized  leadership  development  and  education 
(LEAD).  Although  USU  has  developed  medical  leadership  for  over  thirty  years,  the  LEAD 
program  was  established  1  October  2014  to  focus  on  formal  leadership  education,  development, 
and  assessment.  In  addition  to  traditional  classroom  instruction  and  practical  exercises,  the 


LEAD  program  also  includes  original  research  designed  to  assess  students,  provide  program 
evaluation,  and  develop  medical  leadership  theory.  The  USU  LEAD  program  is  discussed  in 
greater  detail  in  the  following  background  section,  yet  it  is  necessary  to  frame  this  doctoral 
project  as  part  of  the  continued  effort  to  develop  meaningful  leadership  scholarship  that 
addresses  the  critical  need  of  medical  leadership  assessment  as  explained  above.  Additionally, 
this  project  will  describe  how  the  development  and  assessment  of  medical  leadership  is  not 
limited  to  the  military,  but  is  a  growing  need  throughout  medical  schools  and  healthcare  systems 
throughout  the  United  States. 

The  following  sections  include  relevant  background  of  leadership  in  military  healthcare, 
leadership  assessment,  and  the  role  of  the  Uniformed  Services  University  (USU)  in  medical 
leadership  development.  This  project’s  qualitative  methods  are  explained  through  the  use  of  the 
Delphi  method  -  a  structured  communication  technique  used  to  develop  expert  consensus  when 
little  about  a  topic  is  currently  known  (16).  The  project’s  results  are  then  presented,  concluding 
with  a  presentation  of  the  new  Military  Medical  Leadership  Assessment  (MMLA)  instrument 
and  a  discussion  about  how  the  new  instrument  may  enhance  the  quality  of  healthcare  within 
military  and  civilian  settings. 
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Background 

To  understand  the  scope  of  this  project  and  the  importance  of  military  medical  leadership, 
the  following  contains  a  brief  discussion  about  the  military  health  system,  current  strategic-level 
challenges  within  the  MHS,  the  importance  of  Joint-Service  healthcare,  and  the  role  of  the 
Uniformed  Services  University  (USU).  Each  of  the  following  background  sections  are  intended 
to  highlight  how  uniformed  medical  leadership  has  important  consequences  on  national  policy, 
strategic  military  operations,  and  the  future  of  the  Uniformed  Services  University.  More 
importantly,  the  background  that  follows  is  intended  to  frame  the  importance  of  assessing 
medical  leadership  as  an  essential  competency  in  military  medicine  from  the  patient  level  to 
national  healthcare  policy. 

The  Military  Health  System  (MHS) 

The  current  U.S.  military  health  system  (MHS)  is  one  of  the  largest  healthcare  service 
providers  in  the  world.  As  of  2014,  the  MHS  includes  56  hospitals,  365  clinics,  and  more  than 
58,000  civilians  and  86,000  uniformed  personnel  to  provide  comprehensive  healthcare  to  9.7 
million  beneficiaries  (94).  Additionally,  the  MHS  provides  millions  of  dollars  in  translational 
research,  drafts  military  healthcare  policies,  and  provides  compensation  to  civilian  providers  who 
supplement  healthcare  for  the  nearly  10  million  beneficiaries  (94).  These  beneficiaries  include 
active  duty  service  members,  uniformed  retirees,  and  their  dependents  (as  compared  with  the  VA 
which  provides  care  for  all  veterans,  including  those  without  military  retirement  benefits).  The 
nonpartisan  Congressional  Budget  Office  (CBO)  notes  that  the  budget  for  the  MHS  in  fiscal  year 
2013  was  more  than  $50  billion  and  accounts  for  more  than  10%  of  the  overall  Department  of 
Defense  (DoD)  budget.  The  CBO  predicts  that  by  2030,  the  MHS  will  account  for  15%  of  the 
overall  DoD  budget  (137).  The  care  received  by  beneficiaries  covers  virtually  any  healthcare 
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procedure,  assessment,  or  intervention  with  annual  out-of-pocket  expenses  capped  at  $1000  for 
active  duty  and  reserve  families,  and  $3000  for  retirees  (94).  Additionally,  the  MHS  also 
supports  all  healthcare  requirements  for  every  military  mission,  operation,  and  deployment 
anywhere  in  the  world.  The  MHS  is  without  peer  in  terms  of  the  number  of  individuals  provided 
services,  the  variety  of  healthcare  services  offered,  and  the  physical  environments  where  the 
MHS  routinely  operates  -  from  McMurdo  Station,  Antarctica,  to  the  American  military 
astronauts  living  aboard  the  International  Space  Station.  The  unparalleled  scope  of  operations 
within  the  MHS  coupled  with  an  uncertain  financial  future  underscores  the  need  for  leaders  who 
are  responsible,  accountable,  and  adaptive.  This  sentiment  is  similarly  echoed  among  civilian 
healthcare  providers  who  note  that  addressing  the  “leadership  gap,”  or  the  lack  of  formal 
leadership  development  and  assessment  programs,  is  critically  important  to  improve  healthcare 
efficiency  and  stem  rising  healthcare  costs  (31).  The  financial  and  political  considerations  of 
providing  high  quality  healthcare  to  millions  of  Warriors  and  their  families  present  additional 
leadership  challenges  that  warrant  consideration. 

Financial  and  Political  Challenges 

Despite  the  impressive  array  of  healthcare  services,  beneficiaries,  and  missions,  the  MHS 
is  certainly  not  without  challenges  inherent  within  any  large  healthcare  organization.  The  most 
obvious  is  cost.  Healthcare  costs  are  increasing  throughout  the  United  States  and  the  MHS  is 
certainly  no  exception.  The  lower  patient  copays,  high  quality  care,  and  relative  ease-of-access 
were  reasons  the  CBO  gave  to  explain  the  significantly  higher  use  of  both  inpatient  and 
outpatient  care  among  MHS  beneficiaries  when  compared  with  others  with  private  healthcare 
insurance  (94).  As  MHS  beneficiaries  grow  older,  live  longer,  and  the  full  healthcare  costs 
(veterans’  physical  and  psychological  injuries  and  financial  costs)  of  the  current  Global  War  on 
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Terror  (GWOT)  are  assessed,  there  are  strong  doubts  among  lawmakers  that  the  current  MHS 
will  be  able  to  provide  the  same  level  of  care  at  current  costs  (32).  In  response,  both  the 
President  and  the  Secretary  of  Defense  have  suggested  modest  copay  increases  to  meet  rising 
health  care  costs  in  the  MHS.  Such  suggestions  have  been  met  with  strong  negative  reactions 
from  lawmakers,  many  of  whom  believe  that  service  members  were  “promised”  free  healthcare 
in  exchange  for  military  service  (94).  Public  opinion  from  all  sides  of  the  political  spectrum 
remains  solidly  against  any  moves  to  reduce  or  curtail  veteran  health  benefits  (106).  The 
demands  of  an  uncertain  and  unpredictable  political  environment  underscore  the  need  for 
adaptable  leaders  of  high  integrity  who  are  able  to  be  held  accountable  to  Congress,  the  DoD, 
and  the  American  people.  The  DoD  has  begun  to  initiate  modest  changes  aimed  at  encouraging 
military  healthcare  leaders  to  generate  creative  solutions  to  political  and  financial  challenges. 
Perhaps  these  changes  are  most  clearly  seen  in  the  renewed  push  for  joint-service  healthcare 
leadership. 

In  2013  the  Secretary  of  Defense  ordered  the  Deputy  Secretary  for  Health  Affairs  to 
consolidate  a  number  of  healthcare  business  and  policy  operations  (including  the  Office  of 
Transformation  mentioned  above)  into  a  joint-service  organization  called  the  “Defense  Health 
Agency”  (DHA).  The  mission  of  DHA  is  to  serve  as  the  executive  health  agency  for  the  MHS  to 
achieve  medical  readiness,  improve  health,  enhance  the  experience  of  care,  and  lower  healthcare 
costs.  Despite  the  best  intentions  of  the  DHA  to  streamline  healthcare  costs  and  improve 
healthcare  delivery,  the  nascent  agency  faces  years  of  resistance  to  unify  efforts  among  the 
medical  departments  of  the  constituent  uniformed  services  (Army  Medical  Department,  Navy 
Bureau  of  Medicine  and  Surgery,  Air  Force  Medical  Service,  Public  Health  Service)  (89;  167). 
The  latest  attempt  at  a  unified  military  medical  command  was  met  with  reluctance  from  some 
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high-ranking  military  medical  leaders  (149).  Accordingly,  each  uniformed  service  has 
developed  complex,  expensive,  and  largely  redundant  systems  to  deliver  care  to  their  MHS 
beneficiaries  (94;  137).  As  a  compromise,  the  DHA  was  directed  to  assume  direct  control  over 
all  healthcare  facilities  within  the  Washington,  D.C.,  area  under  a  new  umbrella  designation 
termed  “Joint-Task  Force  Capital  Medicine  (JTF-CAPMED).”  What  makes  JTF-CAPMED 
unique  is  that  each  of  the  three  DoD  services  represented  within  the  MHS  has  a  significant 
healthcare  footprint  within  the  vicinity  of  Washington,  D.C.,  and  answers  to  a  single  chain  of 
command  that  consists  of  healthcare  leaders  from  the  Army,  Navy,  Air  Force,  Public  Health 
Service,  and  DoD  Civilians.  In  many  ways,  the  new  JTF-CAPMED  represents  a  smaller  version 
of  what  was  originally  envisioned  by  the  DoD:  multiple  services  working  in  unison  without  the 
inter-service  rivalries  and  costly  redundancies  that  have  plagued  past  attempts  to  create  joint 
uniformed  healthcare.  The  implications  for  healthcare  leadership  reach  beyond  financial  and 
political  costs.  The  21st  Century  uniformed  healthcare  leader  must  be  culturally  sensitive  and 
adaptive  to  leading  in  organizations  and  environments  previously  unknown.  Throughout  the 
DoD,  a  new  culture  of  joint  healthcare  has  emerged  having  strong  influence  on  how  uniformed 
healthcare  leaders  are  developed  and  assessed. 

Joint  Uniformed  Healthcare 

Despite  the  misgivings  of  some  past  senior  military  and  political  leadership,  the  support 
for  joint  military  medicine  is  strong.  A  number  of  analyses  from  senior  military  policy 
institutions  (46;  76;  89;  167),  and  the  Congressional  Budget  Office  (137)  suggest  that  joint 
military  medicine  may  save  nearly  half  a  billion  dollars  a  year  by  eliminating  redundant  systems 
and  cutting  back  waste.  In  response  to  the  reluctance  to  create  a  unified  medical  command, 
LTG(R)  Eric  Schoomaker  (149)  stated  that  there  was  “merit  in  considering  the  most  effective 
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and  efficient  command  structures  to  support  the  strategic  goals  of  the  military  health  system,  the 
services  and  the  combatant  commanders.”  Fiscal  and  political  realities  are  forcing  the  uniformed 
services  to  rapidly  unify  healthcare  efforts  in  ways  that  only  a  decade  ago  were  considered 
farfetched.  Perhaps  the  most  visible  example  is  the  merger  of  Walter  Reed  Army  Medical 
Center  and  National  Naval  Medical  Center  to  create  one  of  the  largest  military  hospitals  in  the 
world  —  Walter  Reed  National  Military  Medical  Center  (WRNMMC).  Another  example 
involving  the  Air  Force  and  Army,  Brooks  Army  Medical  Center  and  Wilford  Hall  Medical 
Center  merged  to  create  the  San  Antonio  Military  Medical  Center  (SAMMC).  Currently,  a  Navy 
thoracic  surgeon  may  lead  a  healthcare  team  composed  of  Army  nurse  anesthetists  in  a  joint 
military  treatment  facility  (MTF)  that  would  have  been  difficult  to  imagine  just  ten  years  ago. 
This  joint  military  medical  model  is  even  more  marked  in  enlisted  medical  training,  which 
consolidated  into  a  single  tri-service  training  program  at  Ft  Sam  Houston,  Texas  in  2011.  Today, 
the  newly  minted  Navy  medical  corpsman  serving  on  a  ballistic  missile  submarine  deep  in  the 
Atlantic  completed  initial  medical  training  with  the  freshly  arrived  Army  medic  located  high  in 
the  Hindu-Kush  Mountains  of  Afghanistan.  Additionally,  new  military  treatment  facilities  such 
as  Ft  Belvoir  Community  Hospital  (FBCH)  are  designed  from  inception  to  be  medical 
partnerships  between  the  different  services  and  among  civilian  community  leaders. 

In  this  new  collaborative  environment,  the  effective  uniformed  healthcare  leader  must 
inspire  and  lead  diverse  groups  of  people  working  towards  common  goals  despite  different 
organizational  cultures,  biases,  experiences,  and  varying  levels  of  motivation.  Developing  such 
a  leader  to  adapt  and  thrive  in  this  environment  requires  new  approaches  to  training, 
teaching,  and  assessment  from  a  Joint-Service  healthcare  perspective.  However,  as  the 
Rand  study  highlighted  above,  no  definition  of  Joint-Service  healthcare  leadership  nor  an 
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instrument  to  assess  Joint-Service  healthcare  leadership  currently  exist.  Fortunately,  within  JTF- 
CAPMED,  there  is  a  long-established  uniformed  academic  healthcare  institution  that  is  uniquely 
capable  of  directly  answering  the  gaps  in  Joint-Service  healthcare  leadership  development:  The 
Uniformed  Services  University  (USU). 

Uniformed  Services  University  (USU) 

Created  by  an  act  of  Congress  in  1972,  USU  is  the  only  federal  medical  school  in  the 
United  States.  Sometimes  called  the  “West  Point  for  Doctors,”  USU  trains  uniformed  student 
physicians,  advanced  nurses,  and  medical  scientists  in  the  four  uniformed  healthcare  agencies 
(Army,  Navy,  Air  Force,  and  Public  Health  Service).  In  addition  to  receiving  an  accredited 
education  in  medicine  or  medical  science,  uniformed  students  at  USU  also  receive  joint-service 
and  service-specific  military  leadership  training  designed  to  develop  them  as  military  medical 
officers.  When  compared  to  other  sources  of  uniformed  providers  (Health  Professions 
Scholarship  Program  [HPSP],  direct  commission),  USU  graduates  serve  longer  in  the  MHS  and 
often  occupy  senior  levels  of  MHS  leadership  (49).  Since  1972,  twenty  one  graduates  of  USU 
have  attained  the  rank  of  flag  officer,  including  two  uniformed  Surgeon  Generals  (154). 

Prior  to  2014,  USU’s  Department  of  Military  and  Emergency  Medicine  (MEM)  provided 
training  in  leadership  development  through  formal  classroom  instruction  on  the  art  of  military 
briefing,  military  medical  history  lectures,  and  Operation  Bushmaster  -  a  week-long  field 
exercise  where  senior  medical  students  and  advanced  practice  nurses  practice  field  medicine  and 
small  unit  tactics  at  Ft.  Indiantown  Gap,  PA.  Although  these  activities  met  the  standard  for 
leadership  education  and  development  throughout  the  first  several  decades  of  USU’s 
establishment,  the  numerous  political,  financial,  and  leadership  challenges  outlined  above  has 
brought  renewed  focus  on  USU’s  important  role  to  develop  and  assess  uniformed  healthcare 
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leaders.  In  a  meeting  with  USU  leadership  in  early  March  of  2014,  the  senior  ranking  civilian 
leader  in  the  MHS,  Assistant  Secretary  for  Health  Affairs  Jonathan  Woodson,  M.D.,  stated 
succinctly,  “ USU  must  lead  and  excel  in  leadership  training ”  (136).  Echoing  this  sentiment  in 
the  University’s  2014-2018  Year  Strategic  Framework  (155),  USU’s  President  Charles  Rice, 
M.D.,  wrote: 

“By  the  end  of  2015,  the  Uniformed  Sendees  University  of  the  Health  Sciences, 
recognized  as  the  preeminent  educational  institution  for  the  creation  of  career  uniformed 
services  health  professionals  and  leaders  prepared  to  serve  the  nation  wherever  and 
whenever  duty  calls...  USU  will  insure  that  each  graduate  is  prepared  with  an  outstanding 
health  education,  inter-professional  health  training,  and  a  deep  and  abiding  commitment 
to  selfless  sendee  and  responsible  leadership ...  ”(136)  [Bold  added] 

The  re-emphasis  on  leadership  development  has  culminated  in  the  establishment  of  the 
USU  Leadership  Education  and  Development  (USU  LEAD)  program  which  is  at  the  core  of  the 
present  doctoral  project. 

USU  Leadership  Education  and  Development  (USU  LEAD) 

In  response  to  Secretary  Woodson  and  President  Rice’s  strategic  vision,  the  Uniformed 
Services  University  undertook  a  number  of  important  steps  and  organizational  changes.  In  the 
fall  of  2014,  retired  Army  Surgeon  General  LTG  (R)  Eric  Schoomaker,  M.D.,  Ph.D,  was 
appointed  as  a  full  professor  in  the  Department  of  Military  and  Emergency  Medicine  (MEM)  and 
granted  the  title  of  Vice  Chair  for  Leadership,  Programs,  and  Centers.  Dr.  Schoomaker,  who  had 
spent  the  academic  year  of  2013  as  a  scholar-in-residence  identifying  areas  to  improve  leadership 
development,  immediately  set  to  work  helping  to  establish  the  USU  Leadership  and 
Development  (USU-LEAD)  program  (136).  Among  the  other  important  changes  within  the 
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University  was  the  primary  academic  affiliation  transfer  of  Neil  E.  Grunberg,  Ph.D.,  a  long- 
serving  member  of  the  Department  of  Medical  and  Clinical  Psychology  (MPS)  and  an  expert  in 
social  and  medical  psychology,  to  MEM.  Dr.  Grunberg  took  on  the  new  role  of  Director  of 
Leadership  Research  and  Development  for  LEAD,  in  addition  to  serving  as  doctoral  dissertation 
adviser  to  this  project,  co-chair  of  the  Bushmaster  Research  Activities  Group  (BRAG),  faculty 
adviser  to  medical  student  leadership  capstone  projects,  and  active  participant  in  activities  related 
to  USU  LEAD. 

Leadership  has  been  identified  as  one  of  the  five  critical  domains  in  the  2014-2015  USU 
Strategic  Framework  -  the  others  being  Education  and  Training,  Research  and  Scholarship, 
National  Security  and  Global  Health  Engagement,  and  Service  (155).  Leadership  development 
has  been  further  sectioned  into  “Leadership  Development”  and  “Thought  Leadership  (155).  This 
new  leadership  curriculum  includes  classroom  instruction,  practical  exercises,  and  student 
scholarship  -  including  three  fourth  year  medical  student  capstone  projects  focused  on  peer 
(student-to-student)  performance  assessment.  The  USU  LEAD  program  also  has  incorporated 
and  improved  existing  leadership  development  exercises  including  Operation  Bushmaster 
(described  in  greater  detail  in  the  following  section)  and  Operation  Gunpowder,  a  two  day 
exercise  developed  for  third  year  student  physicians  that  focuses  on  tactical  combat  casualty  care 
(TC3),  teamwork,  and  communication. 

The  USU  LEAD  team  also  has  started  a  dialogue  with  scholars  at  the  three  DoD  service 
academies  (United  States  Military  Academy  [USMA],  United  States  Naval  Academy  [USNA], 
United  States  Air  Force  Academy  [USAFA]),  and  the  Duke  University’s  “Feagin  Leadership 
Program”  -  a  specialty  track  that  medical  students  at  Duke  University  may  opt  to  take  prior  to 
residency.  In  contrast  with  Duke,  where  leadership  development  is  an  optional  specialty,  every 
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student  physician  at  USU  is  a  unifonned  commissioned  officer  who  will  be  expected  to  be  a 
dynamic  healthcare  leader  in  the  challenging  and  continually  transforming  MHS  described  above 
(81).  Simply,  leadership  development  at  USU  is  a  requirement  for  student  physicians  akin  to 
passing  the  United  States  Medical  Licensing  Exam  (USMLE)  (136). 

The  USU  LEAD  program  has  underscored  the  need  to  identify  what  components  are 
important  for  unifonned  healthcare  leadership  and  how  to  effectively  assess  healthcare 
leadership.  Moreover,  as  outlined  above,  the  need  for  a  model  of  uniformed  healthcare 
leadership,  and  any  subsequent  assessment  instrument,  must  be  created  with  a  Joint-Service 
perspective. 

Assessment  and  USU  LEAD 

The  USU  LEAD  program  is  a  diverse  curriculum  that  involves  traditional  classroom 
education,  small  group  exercises,  and  mentorship  opportunities  between  USU  faculty  and  student 
physicians.  Additionally,  the  LEAD  program  includes  leadership  research  from  medical  students 
conducting  capstone  exercises  and  graduate  students  evaluating  the  efficacy  of  Operation 
Bushmaster  (BRAG),  a  multi-day  field  medical  exercise  for  advanced  student  physicians  and 
nurses  held  annually  at  Ft.  Indiantown  Gap,  PA.  Operation  Bushmaster  is  a  field  medicine  and 
small  unit  leadership  exercise,  allowing  the  fourth  year  students  to  assume  the  roles  of  platoon 
leader,  battalion  surgeon,  and  patient  administrator  in  a  Role-1  (i.e.,  basic  life  support)  tactical 
treatment  setting  (136).  Each  year,  experienced  civilian  and  uniformed  faculty  from  USU  and 
throughout  the  MHS  act  as  evaluators,  assessing  individual  medical  and  healthcare  leadership 
skills.  In  after  action  reports  (AARs)  with  the  LEAD  team,  the  Bushmaster  faculty  has  identified 
the  lack  of  a  user-friendly  Joint-Service  medical  leadership  assessment  instrument  as  a  critical 
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shortage.  The  current  doctoral  project  was  designed  to  address  this  gap  through  the 
accomplishment  of  two  primary  objectives: 

1)  Identify  what  elements  of  healthcare  leadership  are  important  to  develop  in  student 
physicians  at  the  Uniformed  Services  University 

2)  Create  a  military  medical  leadership  assessment  instrument  that  is  easy  to  use  and  is  both 
face  and  content  valid. 

Before  discussing  how  this  assessment  instrument  was  created,  a  historical  perspective  of 
military  medical  leadership  assessment,  relevant  definitions  of  leadership  guiding  the  project, 
and  a  review  of  the  qualitative  methods  used  in  the  development  of  the  Military  Medical 
Leadership  Assessment  (MMLA)  is  necessary. 

Historical  Perspective  of  Medical  Leadership  Assessment 

The  importance  of  leadership  development  in  American  military  healthcare  can  be 
observed  on  virtually  every  level  beginning  with  the  establishment  of  a  “hospital  corps”  by  the 
Continental  Congress  in  1775.  A  full  year  before  the  founding  framers  would  meet  in 
Philadelphia  and  declare  independence  from  Great  Britain,  the  nascent  American  Army  was 
already  in  the  midst  of  its  first  crisis  of  military  healthcare  leadership.  Along  with  the  creation  of 
the  first  regiment  of  physicians  and  surgeons,  the  Continental  Congress  appointed  a  Boston-born 
physician.  Dr.  Benjamin  Church,  as  the  Chief  Physician  and  Director  General.  Church,  a 
member  of  the  Boston  aristocracy,  was  less  than  enthusiastic  about  his  appointment  and 
numerous  complaints  about  his  competence  flooded  GEN  George  Washington’s  besieged 
headquarters  (115).  In  addition  to  his  leadership  deficits,  Dr.  Church  had  amassed  significant 
gambling  debts  (13).  Seeing  no  other  alternative,  Church  turned  to  the  British  forces  for  relief 
and  began  sending  classified  information  about  Continental  forces  arrayed  in  the  Boston  area. 
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One  of  Church’s  ciphered  letters  was  intercepted  by  Colonial  counter-intelligence  agents  and  he 
was  unceremoniously  cashiered  from  his  commission  and  remained  under  house  arrest  for  the 
duration  of  the  war  (73). 

The  following  two  physicians  to  occupy  the  Director  General  position  fared  little  better. 
Dr.  John  Morgan,  who  co-founded  the  College  of  Philadelphia  medical  school,  quit  the  post  after 
incessant  bickering  and  quarrelling  among  Army  physicians  (79).  His  successor,  Dr.  William 
Shippen,  another  well-known  Philadelphia  physician  and  bitter  rival  of  Morgan’s,  lasted  only  a 
few  months  in  his  position  before  being  court  martialed  for  misappropriating  supplies  and  false 
reporting  casualty  numbers  to  Washington’s  staff  (115).  It  was  not  until  a  relatively  unknown 
Irish  American  army  physician  named  John  Cochrane  was  appointed  as  Director  General  in  the 
spring  of  1777,  that  Washington  and  his  line  commanders  began  to  develop  confidence  in  the 
leadership  of  the  Army  Medical  Department  (AMEDD)  (79). 

A  career  Army  officer,  Cochrane  initiated  a  number  of  important  reforms  to  the 
AMEDD,  including  personally  certifying  the  professional  competence  and  quality  of  every 
regimental  surgeon  (163).  Dr.  Cochrane’s  focus  in  assessing  the  military  medical  leadership 
abilities  of  his  subordinate  physicians  was  essential  to  the  very  first  successes  of  the  AMEDD. 
After  the  first  two  years  of  its  existence  was  marked  by  treason,  incompetence,  and  toxic 
leadership,  American  military  medicine  finally  began  to  develop  as  an  organization  that  greatly 
enhanced  the  ability  of  the  Continental  Army  to  victory  in  the  American  Revolution.  Dr. 
Cochrane  personally  visited  and  interviewed  each  regimental  surgeon,  assessing  their  medical 
skills  and  ability  to  support  their  front-line  commanders.  Recognizing  the  leadership  qualities  of 
Dr.  Cochrane,  GEN  Washington  wrote  “I  will  mention. .  .that  I  think  that  Dr.  Cochrane. . .  [is] 
among  the  [finest]  officers  of  the  [medical]  establishment.”  (115).  Although  medicine  during  the 
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Revolution  was  by  modern  standards,  still  a  very  primitive  science,  military  medical  historians 
often  point  to  Cochrane  as  the  best  example  of  military  medical  leadership  during  the  turbulent 
colonial  years  (79).  In  the  subsequent  decades,  military  healthcare  saw  numerous  changes  - 
from  the  standardization  of  medical  school  training  in  the  aftermath  of  the  Civil  War,  to  the 
eradication  of  tropical  yellow  fever  under  the  tireless  efforts  of  Surgeon  General  William  Gorgas 
and  the  development  of  rapid  aeromedical  evacuation  under  Maj  Gen  Malcolm  Grow  (179).  As 
Western  healthcare  rapidly  developed  in  the  later  part  of  the  19th  century,  the  need  to  assess 
quality  military  healthcare  leadership  has  increased  alongside  the  significant  and  rapid  advances 
in  military  medicine  (140). 

It  should  be  noted  that  Dr.  Cochrane  and  subsequent  uniformed  healthcare  leaders 
throughout  history  largely  relied  on  their  own  intuition  and  personal  judgment  in  assessing 
military  medical  leadership.  These  healthcare  leaders,  although  famous  for  their  contributions  to 
medicine  within  the  military  and  the  nation,  did  not  possess  a  model  of  healthcare  leadership  or  a 
formal  evaluative  tool.  Even  as  late  as  1990,  the  Department  of  Defense  had  no  objective 
leadership  assessment  instrument  -  healthcare  or  otherwise  (121).  Although  psychological 
assessment  has  been  successfully  used  for  decades  to  help  assign  individuals  to  certain  roles  and 
occupations  within  the  military  and  to  screen  for  psychopathology  among  uniformed  personnel, 
the  field  of  uniformed  leadership  assessment  has  lagged  far  behind  (34).  As  the  Rand  study 
(103)  and  DoD  Leadership  (60)  have  stated  numerous  times  above,  the  gaps  in  leadership 
assessment  are  especially  seen  within  the  military  healthcare  system  (MHS).  The  present 
doctoral  project  seeks  to  address  this  shortcoming  by  creating  a  brief  military  medical  leadership 
assessment  (MMLA)  instrument  based  in  scientific  theory  through  qualitative  methods  described 
in  Chapter  3. 
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Theoretical  Perspective 

Brief  History  of  Leadership  Theory 

Although  the  present  project  aims  to  develop  a  new  military  medical  leadership 
assessment  instrument,  it  is  necessary  to  ground  the  proposal  in  appropriate  scientific  theory. 
Classical  texts  on  leadership  include  Homer’s  Iliad,  the  Babylonian  Talmud,  the  Tao  Te  Ching  of 
ancient  China,  as  well  as  the  ancient  Vedic  Bhagavad  Gita  (128).  In  modem  times,  the  first 
major  scientific  investigation  into  leadership  theory  was  in  1939,  when  German  Jewish  refugee 
mathematician-turned-social  psychologist  Kurt  Lewin  explored  aggressive  behavior  in  school 
children  that  had  been  assigned  to  three  different  groups  -  democratic,  authoritarian,  and  laissez- 
faire  (110).  Lewin  and  his  colleagues  observed  that  children  with  a  democratic  leader  were  less 
aggressive  and  less  hostile  when  compared  to  the  other  leadership  styles  (69).  Lewin’s  work  was 
particularly  influential  in  subsequent  decades  (148)  with  the  “democratic”  leadership  style 
emerging  as  the  preferred  leadership  style  (71).  The  democratic  style  of  leadership  emphasizes 
mutual  respect  between  subordinates  and  leaders  and  provides  opportunities  for  subordinates  to 
communicate  their  views  and  participate  in  leader  decisions  (70),  or  what  Ferguson  (70) 
describes  as  “freedom  with  order.”  This  leadership  style  contrasts  starkly  with  the 
“authoritarian”  (i.e.,  order  with  no  freedom)  or  “laissez  faire”  (i.e.,  freedom  with  no  order)  styles. 
Throughout  the  1950s  and  1960s,  the  democratic  approach  was  endorsed  as  the  preferred  method 
for  conducting  group  psychotherapy  (64),  family  therapy  (63),  classroom  instruction  (65),  and 
child  rearing  (23). 

Despite  the  influence  of  Lewin’s  “Big  Three”  on  leadership  research,  the  typology  has 
been  critiqued  in  the  decades  following  the  modern  American  civil  rights  movement  (38). 
Kippenburger  (102)  and  Burnes  (37)  criticized  Lewin’s  approach  as  too  rigid  and  linear,  and 
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asserted  that  leadership  of  organizations  is  an  open-ended,  fluid,  and  dynamic  process. 

Buchanan  (36)  observed  that  Lewin’s  categories  ignore  the  influence  of  power  and  politics 
within  organizations,  including  racism  and  intolerance.  Hardy  (84)  echoed  the  hierarchical 
concerns  of  Kippenburger  (59)  and  Burnes  (17)  by  suggesting  that  changes  in  an  organization  are 
often  wholly  the  product  of  the  subordinates  and  may  not  require  a  leader  at  all.  Even  Lewin’s 
daughter  Miriam  Lewin,  Ph.D.,  an  established  social  psychologist  in  her  own  right,  argued  that 
the  conceptualization  of  “democratic  leadership”  is  poorly  defined  (111). 

Several  decades  of  research  into  leadership  theory  have  developed  new  theories  to 
address  the  criticisms  outlined  above.  These  theories  include  the  Leadership  Trait  Approach  (97; 
170),  Skills  Approach  (134),  Style  Leadership  (29),  Situational  Leadership  (30),  and  Relational 
Theories.  Relational  Theories  emphasize  the  interpersonal  dynamics  between  a  leader  and 
subordinates  in  a  manner  similar  to  the  original  Lewinian  conceptualization,  albeit  with  far 
greater  emphasis  on  leadership  as  a  fluid  construct  that  Cunliffe  (52)  describes  as  “post-heroic 
[leader].”  Relational  theories  of  leadership  include  Servant  Leadership  (78),  Authentic 
Leadership  (11),  and  Transformational  Leadership  (39). 

Importance  of  Relational  Theories  in  Military  Medicine 

The  present  project  did  not  compare  and  contrast  these  many  leadership  theories  in  situ. 
Rather,  these  theories  are  presented  to  acknowledge  that  the  field  of  leadership  research  is  vast 
and  each  style  has  unique  advantages  and  disadvantages  -  especially  in  terms  of  assessment  in 
military  medicine.  Of  the  categories  presented,  the  creation  of  the  MMLA  in  this  project  is 
largely  founded  in  relational  leadership  theory.  Relational  theories  emphasize  the  importance  of 
the  relationship  between  leaders  and  followers  as  the  essential  element  to  organizational  success 
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and  leadership  effectiveness  (105).  A  relational  leadership  theory  is  particularly  useful  to 
military  medicine  for  the  following  reasons: 

a.  A  relational  leadership  perspective  is  framed  within  the  context  of  human  interactions. 
As  medicine  is  inherently  an  interaction  between  human  patients  and  providers,  relational 
leadership  is  well  suited  for  leadership  training  in  medical  schools  (150)  and  healthcare 
practice  (168). 

b.  The  military  is  also  an  institution  largely  defined  by  human  relationships.  Relational 
leadership  elements  have  high  correlation  with  mission  success  in  elite  units  (85)  and 
military  units  in  combat  (58). 

Transformational  Leadership  and  Military  Medicine 

Of  the  relational  leadership  theories,  the  background  of  this  project  is  further  defined  and 
focused  with  Transformational  Leadership  Theory  (TLT).  TLT,  which  was  first  proposed  by 
Bumes  (39)  and  later  updated  by  Bass  (19),  asserts  that  effective  leadership  is  accomplished 
through  “transformative  relationships”  that  raises  the  motivation  of  both  leader  and  follower. 

The  transformative  relationship  between  leader(s)  and  follower(s)  are  characterized  by  four 
major  domains:  charisma,  inspiration,  intellectual  stimulation,  and  individualized  consideration 
(20). 

Elements  of  TLT  have  been  positively  correlated  to  predicting  the  mission  success  of 
military  units  (22).  Empirical  reviews  of  TLT  in  healthcare  settings  show  that  transformative 
relationships  between  leaders  and  followers  improves  the  wellbeing  of  healthcare  providers 
(181)  and  clinical  effectiveness  of  the  healthcare  organization  (189).  Especially  important  to  the 
present  proposal,  TLT  has  been  observed  to  promote  resiliency  in  military  units  which  may 
mitigate  the  effects  of  combat  stress  and  trauma  (18)  and  to  counter  the  effects  of  toxic 
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leadership  (18;  42;  62).  TLT  is  most  commonly  assessed  by  the  Multifactor  Leadership 
Questionnaire  (MLQ),  a  45-item  self-report  assessment  designed  for  subordinates  to  rate  a  leader 
and  for  the  leader  to  rate  themselves  (21).  The  MLQ,  which  is  the  most  widely  used 
commercially-available  leadership  assessment,  produces  a  profile  that  assesses  how  a  leader  rates 
on  the  four  major  domains  of  TLT  (133). 

TLT  and  the  FourCe  Model 

Transformational  leadership  is  not  without  critics.  Tracey  and  Hinkin  (175)  note  that  the 
four  major  domains  of  TLT  described  above  have  significant  overlap.  A  related  criticism  from 
Tejada,  Scandura,  and  Pillai  (172)  notes  the  subscales  on  the  MLQ  are  highly  correlated  to  each 
other,  and  even  to  less  effective  leadership  styles.  Additionally,  TLT  does  not  directly  address 
context  or  the  importance  of  group  dynamics,  which  is  especially  important  in  terms  of 
leadership  in  military  healthcare.  Group  dynamics  have  been  an  essential  component  of 
scholarly  work  into  military  leadership  since  Lewin’s  (109)  landmark  1947  article  that  urged 
psychologists  and  social  scientists  to  conduct  experiments  that  examine  how  individual  behavior 
is  influenced  by  the  physical  or  perceived  presence  of  others.  Dion  (59)  places  special  emphasis 
on  the  social  and  psychological  context  of  a  group  as  a  major  predictor  of  achieving  group  and 
individual  goals.  Other  elements  of  context  important  to  successful  leadership  and 
organizational  success  includes  group  norms,  a  shared  organizational  culture,  and  task- 
cohesiveness  (130).  Drawing  lessons  from  observing  various  group  therapies,  Yalom  (190) 
asserted  that  group  climate  mediates  the  relationship  between  leadership  and  positive  outcomes. 

Recognizing  these  gaps  in  TLT,  Callahan  and  Grunberg  (80)  considered  many  leadership 
models  and  proposed  a  “new”  model  that  draws  on  the  strengths  of  TLT  while  addressing  the 
shortcomings  especially  in  terms  of  physical,  psychological  and  social  context.  This  model, 
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called  the  FourCe’s  (a  mnemonic  that  plays  on  the  word  “Force”)  model,  categorizes  leadership 
into  one  of  four  major  domains: 

•  Character:  Responsibility,  integrity,  confidence,  humility,  high  emotional 
intelligence  (EQ),  empathy 

•  Competency:  Transcendent  leadership  skills  and  expertise  determined  by  role  and 
specialty 

•  Context:  Physical,  psychological,  social  environment,  situations,  stress 

•  Communication:  Verbal  and  non-verbal;  sending  and  receiving  information. 

To  illustrate  the  point  of  the  FourCe  model  in  military  healthcare  settings,  the 

communicative  style  of  a  neurosurgeon  directing  his  team  in  the  operating  room  at  Walter  Reed 
National  Military  Medical  Center  (WRNMMC)  may  be  vastly  different  from  a  physician’s 
assistant  directing  medics  to  provide  emergency  care  under  fire  in  Afghanistan.  Both  scenarios 
require  transcendent  leadership  skills  and  rely  on  the  character  of  the  leader  to  appropriately 
motivate  subordinates  to  achieve  mission  critical  goals.  Where  the  FourCe  model  expands  on 
TLT  is  raising  awareness  of  a  leader  to  the  importance  of  the  motivation  of  subordinates  in  vastly 
different  physical,  social  and  psychological  contexts.  The  creation  of  the  MMLA  is  meant  to 
bridge  the  FourCe  model  with  the  USU  LEAD  curriculum  by  providing  direct  feedback  on  the 
individual  student  level  and  indirect  program  evaluation.  The  MMLA  and  potential  impact  on 
advancing  the  FourCe  model  is  discussed  in  greater  detail  in  Chapter  5. 

FourCe  Model  and  Personal,  Interpersonal,  Team,  &  Organization  (PITO) 

The  final  component  of  the  theoretical  background  necessary  to  frame  the  MMLA  is  the 
longitudinal  PITO  outline  that  complements  the  LourCe  model.  PITO,  rather  than  a  model  of 
military  medical  leadership,  is  a  leadership  training  plan  utilized  in  the  USU  LEAD  program. 
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Originally  developed  at  the  United  States  Air  Force  Academy  (USAFA),  PITO  guides  the  USU 
LEAD  curriculum  by  increasing  leadership  complexity  and  responsibility  over  the  four  years  of 
pre-residency  education  (136).  In  the  earliest  phase  of  leadership  development,  the  goal  is  to 
raise  a  student  physician’s  personal  self-awareness  concerning  beliefs,  biases,  and  individual 
leadership  style  (Personal).  This  stage  is  followed  by  focusing  leadership  development  on 
promoting  healthy  and  productive  interpersonal  interactions  including  effective  communication 
and  emotional  intelligence.  In  the  final  phase  of  PITO  conducted  at  USU,  the  student  physician 
begins  to  learn  about  leading  small  teams,  group  dynamics,  and  basic  principles  of  social 
psychology  relevant  to  healthcare  leadership.  Organizational  leadership  is  introduced  in 
medical  school  but  typically  occurs  in  the  senior  residency  years.  The  PITO  framework  is 
introduced  above  to  highlight  how  the  MMLA  is  intended  to  be  an  instrument  that  may  be  used 
over  time  and  has  applicability  to  each  stage  of  uniformed  student  physician  leadership 
development.  From  this  theoretical  basis,  definitions  of  leadership  relevant  to  the  assessment  of 
military  medical  leadership  follow. 

Definitions 

Numerous  definitions  of  what  constitutes  leadership,  healthcare,  and  leadership  within 
healthcare  abound.  To  ensure  scientific  rigor,  the  present  project  must  first  define  the  most 
important  terms  and  concepts  related  to  military  medical  leadership  and  leadership  assessment. 

In  introducing  the  definitions  used  for  the  present  project,  it  is  important  to  acknowledge  that  the 
perspective  used  hereinafter  stems  from  the  psychosocial  perspective  in  addition  to  the  lens  of 
Transformational  Leadership  outlined  above.  These  definitions  will  be  re-examined  in  Chapter  5 
as  they  relate  to  the  Military  Medical  Leadership  Assessment  (MMLA). 
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Popularized  by  developmental  psychologist  Erik  Erikson  in  1950  ,  the  Psychosocial 
theory  conceptualized  human  development  in  eight  separate  stages  defined  by  the  interaction  of 
psychological  and  social  factors  (69).  Originally  a  human  development  theory,  the  Psychosocial 
theory  has  influenced  major  theories  of  behaviors,  cognitions,  and  motivations  (48;  90).  These 
theories  include  Field  Theory  (108),  Group  Dynamics  (44),  the  Transtheoretical  Model  (152),  the 
contemporary  Health  Belief  Model  (95),  the  Theory  of  Reasoned  Action  (129)  and  formed  the 
core  of  the  Biopsychosocial  Model  (68). 

Additionally,  the  definitions  of  leadership  and  related  concepts  in  academic  work  is  often 
defined  by  the  inherent  professional  biases  and  perspective  of  the  researcher(s);  political 
scientists,  sociologists,  and  even  religious  scholars  all  purport  different  definitions  of  leadership 
(6).  The  aim  of  the  present  project  is  not  to  discredit  or  dismiss  the  decades  of  research  into 
leadership,  its  components,  or  its  various  definitions.  It  is  important,  however,  to  operationally 
define  important  concepts  and  terms  for  the  present  project  in  a  manner  that  has  special  utility  for 
healthcare  and  more  specifically,  uniformed  healthcare. 

Leadership;  The  enhancement  of  behaviors,  cognitions,  and  emotions/motivations  to  achieve 
goals  that  benefit  individuals  and  groups  (80). 

Few  definitions  have  been  as  hotly  and  passionately  debated  than  leadership.  Within 
academic  literature,  the  definition  of  leadership  has  gone  through  many  evolutions  -  from 
focusing  on  the  individual  traits  (i.e.,  the  physical  strength,  attractiveness,  personality  of  the 
leader)  to  being  defined  by  the  various  outcomes  generated  by  a  leader’s  actions  or  inactions 
(i.e.,  number  of  missions  accomplished,  number  of  budget  dollars  saved,  number  of  patient 
safety  violations)  (171).  It  is  important  to  note  that  leadership  can  be  viewed  largely  as  a  social 


28 


construct  that  is  influenced  and  shaped  by  the  contemporary  physical,  social,  and  psychological 
environment  of  an  individual  or  group  (186). 

The  definition  of  leadership  presented  above  stems  from  the  FourCe  model  developed  at 
USU  as  an  extension  of  TLT.  Essential  to  the  definition  of  leadership  introduced  for  the  present 
project  is  the  incorporation  of  the  three  major  components  of  modern  psychology  (behaviors, 
thoughts,  and  motivations)  to  link  the  relatively  little  known  about  the  assessment  of  military 
medical  leadership  to  psychosocial  theory.  The  definition  of  leadership  presented  in  this  project 
also  draws  from  Transformational  Leadership  Theory  (TLT)  and  the  LourCe  model  by 
emphasizing  the  enhancement  of  motivations  and  the  group  as  essential  to  goal 
accomplishment.  It  is  important  that  as  an  assessment  instrument,  the  MMLA  is  able  to  assess 
healthcare  leadership  in  terms  of  the  rated  individual’s  motivational  drive,  and  how  well  the 
leader  is  able  to  build  motivation  in  others.  This  theme  of  motivation  enhancement  by  a  leader 
is  preserved  in  the  following  definitions  of  Medical  Leadership  and  Military  Medical  Leadership. 
Medical  Leadership:  The  enhancement  of  behaviors,  cognitions,  and  emotions/motivations  to 
achieve  goals  that  benefit  individuals  and  groups  within  the  healthcare  team,  promote  patient 
safety  and  health,  and  foster  public  health  and  national  wellbeing. 

Until  recently,  little  peer-reviewed  literature  addressed  medical  leadership  (45).  Critics 
point  to  the  emergence  of  managed  care  in  the  1970s  that  began  to  transform  the  role  of 
physicians  (and  other  doctoral-level  providers)  out  of  traditional  leadership  roles,  ceding  them  to 
business  executives,  attorneys,  and  non-medical  managers  (107).  In  the  past  ten  years,  however, 
a  renewed  interest  in  medical  leadership  has  seen  the  emergence  of  medical  leadership  training 
programs  at  medical  schools  (e.g.,  Columbia,  Harvard,  Duke,  University  of  Southern  Llorida, 
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etc.),  and  a  greater  number  of  articles  appearing  in  respected  journals  ( British  Medical  Journal , 
Harvard  Business  Review,  etc.). 

As  noted  above,  the  definition  of  leadership  continues  to  be  hotly  debated.  In  contrast, 
relatively  little  (debate  or  otherwise)  addresses  the  definition  of  medical  leadership  (45).  For  the 
creation  of  the  MMLA,  the  definition  of  medical  leadership  includes  the  “major  three” 
components  of  cognitive-behavioral  psychology  and  links  them  with  goals  of  the  healthcare 
team,  patient  outcomes,  and  national  health.  The  definition  is  crafted  in  such  a  manner  as  to 
instill  the  MMLA  with  appropriate  psychological  science  with  the  near-universal  healthcare 
values  of  beneficence  and  non-maleficence  (i.e.,  “first,  do  no  harm”)  and  the  responsibility  of 
healthcare  professionals  that  span  from  the  individual  patient  to  the  entire  nation. 

It  is  also  important  to  note  that  the  term  “medical  leadership”  as  defined  above  may  not 
be  exclusively  limited  to  Western  Medical  Model-based  allopathic  medicine.  Although 
“healthcare  leadership”  may  be  a  more  accurate  description,  the  present  project  will  primarily 
examine  developing  physician  leaders  within  the  military  healthcare  system  (MHS)  where  the 
term  “medical”  is  a  more  culturally  relevant  term  for  the  MMLA  in  a  uniformed  medical  setting. 
Military  Medical  Leadership:  The  enhancement  of  behaviors,  cognitions,  and  emotions/ 
motivations  to  achieve  gocds  that  benefit  individuals  and  groups  within  the  military  healthcare 
system  (MHS),  foster  the  health  of  Warriors  and  their  families,  and  supports  the  line  Command 
mission. 

As  virtually  nothing  exists  in  the  formal  definition  or  assessment  of  Military  Medical 
Leadership  (MML),  the  definition  presented  above  is  novel.  As  with  the  preceding  definitions, 
psychological  science  is  linked  with  the  more  specific  elements  of  leadership  in  medical  settings. 
The  unique  elements  of  the  presented  definition  include  identifying  the  military  health  system 
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(MHS)  as  the  primary  environment  as  well  as  Warriors  and  their  families  as  the  critical 
population  served  by  military  medical  leaders.  The  MHS  can  broadly  be  described  as  the  system 
within  the  Department  of  Defense  (DoD)  that  provides  healthcare  to  active  duty  and  retired 
uniformed  members  and  their  dependents  (27).  The  term  “military”  refers  to  not  only  the  four 
major  branches  represented  by  the  DoD  (Army,  Navy,  Air  Force,  Marine  Corps),  but  also  the 
uniformed  Public  Health  Service  (PHS),  uniformed  National  Oceanic  and  Atmospheric 
Administration  (NOAA),  and  the  Coast  Guard.  “Warrior”  refers  to  any  uniformed  member  of 
these  organizations  that  has  access  to  the  MHS.  This  definition  is  reflected  in  the  project’s 
purposeful  selection  of  participants  representing  multiple  uniformed  services,  ensuring  that  the 
MMLA  may  be  face  and  content  valid  in  Joint-healthcare  settings  such  as  USU. 

Responsibility  to  support  the  Commander’s  mission  is  perhaps  the  most  distinguishing 
trait  of  military  medical  leadership.  Unlike  civilian  medical  leadership,  medical  leadership  in  the 
military  has  the  legal  obligation  to  support  the  Commander’s  military  mission  through  medical 
science  and  practice.  The  term  “line  Commander”  is  purposefully  vague  as  it  can  refer  to  all 
levels  of  military  leadership  -  including  Non-Commissioned  Officers  (NCOs).  Similarly,  the 
broad  scope  of  command  included  in  the  presented  definition  covers  all  uniformed  services 
missions-  from  the  tactical/  individual  unit  level  to  the  national/strategic  policy  level.  The  study 
participants  included  both  line-Commanders  (i.e.,  non-medical)  and  line-NCOs  from  multiple 
services. 

It  is  important  to  highlight  that  the  present  research  project  focused  on  the  process  of 
leadership  development  rather  than  defining  who  meets  the  definition  of  leader.  However,  based 
on  the  definition  of  Military  Medical  Leadership  above,  a  Military  Medical  Leader  is  any 
uniformed  healthcare  professional  who  enhances  the  behaviors,  cognitions,  and  motivations  of 
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followers  to  achieve  goals  that  benefit  individual  and  group  goals,  the  mission  of  the  MHS,  and 
the  mission  of  the  line  Command.  Each  uniformed  student  physician,  the  target  population  for 
the  MMLA,  sufficiently  meets  this  definition  of  Military  Medical  Leader. 

Recent  History  of  Military  Leadership  Assessment 

As  mentioned  above,  no  clear  assessment  of  leadership  within  the  United  States  Military 
existed  prior  to  1990  (121).  Throughout  the  mid-1990s,  the  military  used  a  patchwork  of 
leadership  assessment  instruments  with  relatively  little  validation  of  the  effectiveness  of  these 
measures  (192).  Many  of  these  instruments  reflected  post-Cold  War,  peacetime  military  doctrine 
that  became  largely  irrelevant  after  9/1 1  and  the  subsequent  War  on  Terror  (15).  To  reflect 
changes  in  doctrine  and  to  address  the  lack  of  a  uniform  assessment  instrument,  the  Center  for 
Army  Leadership  (CAL)  at  Lort  Leavenworth,  Kansas,  began  a  large  scale  project  to  develop  a 
new  assessment  tool.  Beginning  in  2005,  researchers  at  CAL  created  questionnaires  designed  to 
survey  Non-commissioned  officers  (NCOs),  staff  officers,  and  commanders  about  leadership 
domains  that  had  previously  been  identified  and  arranged  into  a  model  by  leadership  experts  at 
Lort  Leavenworth.  Of  the  domains  created  by  CAL  experts,  14  were  identified  by  study 
participants  as  having  a  Cronbach’s  alpha  (a>.8)  sufficiently  high  enough  for  retention  in  an 
assessment  instrument.  The  result  of  this  project  was  released  in  2009  as  the  “Multi-Source 
Assessment  and  Feedback  System  (MSAF),”  a  web-based  assessment  instrument  designed  to 
assess  leaders  on  the  domains  identified  by  CAL  in  the  original  study.  The  MSAL  measures  the 
original  domains  using  three  different  assessments: 

I  ADAPT  M:  A  self-report  instrument  intended  for  leaders  to  evaluate  themselves  by 
answering  yes/no  questions  on  46  different  statements.  The  I  ADAPT  M  generates  a 
profile  that  produces  seven  subscales:  crisis  situations;  cultural  adaptability;  work  stress; 


32 


interpersonal  adaptability;  learning  new  tasks;  technologies  and  procedures;  creative 
problem  solving;  and  uncertain  and  unpredictable  work  situations. 

Leadership  Behavior  Scale  2.0  (LBS-2):  An  instrument  that  is  both  self-report  and 
solicits  feedback  from  superiors  and  subordinates.  Respondents  answer  50  items  that 
assess  a  leader  on  10  domains:  Lead  Others,  Lead  by  Example,  Create  a  Positive 
Environment,  Communicate,  Develop  Leaders,  Prepare  Self  to  Lead,  Get  Results,  Extend 
Influence  Beyond  Chain  of  Command,  Builds  Trust  and  Stewards  the  Profession.  Items 
are  Likert-style  questions  ranging  from  “Not  At  All  Effective”  to  “Highly  Effective.” 

The  Team  Personality  Inventory:  An  instrument  that  is  both  self-report  and  solicits 
feedback  from  superiors  and  subordinates.  This  instrument  is  designed  to  raise  self- 
awareness  regarding  8  domains  related  to  effective  team  work:  tough  mindedness; 
resourcefulness,  intellectual  capacity;  leadership  motivation;  achievement  -  seeking; 
interpersonal  tact;  teamwork;  trust. 

The  MSAF  has  a  number  of  limitations  that  this  doctoral  project  addresses.  The  first  is 
that  the  MSAF,  despite  it  being  the  most  widely  researched  and  developed  leadership  assessment 
tool  in  the  military,  is  Army-centric  (87;  88).  The  Navy  is  currently  piloting  a  limited 
assessment-feedback  system  within  the  Surface  Warfare  Community.  The  Air  Force  and  Marine 
Corps  do  not  appear  to  have  a  Service-wide  assessment  instrument.  As  described  several  times 
above,  military  medicine  is  becoming  increasingly  Joint-Service  and  any  assessment  instrument 
should  incorporate  the  unique  perspectives  of  all  services  that  comprise  the  Military  Health 
System  (MHS).  The  theoretical  basis  for  the  MSAF  was  developed  by  polling  military  leaders 
over-represented  by  combat  arms  units  (i.e.,  fewer  [if  any]  medical  providers)  (88). 

Additionally,  little  else  is  known  about  the  original  participants  used  to  validate  the  domains  of 
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the  MSAF  (i.e.,  ethnicity,  gender,  education).  The  MMLA,  in  contrast,  took  participant 
demographics  into  consideration  and  ensured  that  the  project  had  appropriate  representation  from 
women  and  ethnic  minorities.  The  MSAF  is  also  resource  intense,  requiring  at  least  an  hour  to 
complete  and  several  days  more  to  provide  any  feedback.  The  MMLA  instrument  developed  in 
this  research  project  is  brief,  taking  only  a  few  minutes  to  complete.  Because  there  does  not 
appear  to  be  any  further  research  on  the  validity  of  the  MSAF  after  2005  (or  the  MSAF’s 
underlying  theoretical  model),  the  MMLA  may  help  to  gain  a  more  current  perspective  on 
medical  leadership  within  the  uniformed  services.  The  implications  of  the  MMLA  on  medical 
leadership  theory  is  discussed  in  greater  detail  in  Chapter  5. 

Currently,  there  are  several  leadership  assessment  tools  in  addition  to  the 
Multidimensional  Leadership  Questionnaire  (MHQ)  and  MSAF  described  above.  One 
assessment  instrument  is  Dennis  and  Bocarnia’s  (57)  “Servant  Leadership  Assessment 
Instrument,”  a  42-item  instrument  that  measures  leadership  along  five  domains  of  Patterson’s 
(145)  Servant  Leadership  Theoretical  Model.  Another  instrument  commercially  available 
through  Linkage,  Inc.  is  the  “Leadership  Assessment  Instrument,”  which  has  its  roots  in  Bemis’ 
“Revisionist  Leadership  Theory”  (26).  Other  leadership  assessments  stem  from  Feminist 
theories  (28),  Organizational  psychology  (93),  and  even  a  “Business  Ecosystem”  model  (173). 
Some  leadership  assessment  tools  stem  from  an  identified  leadership  model  or  theory  (3,  13,  21), 
whereas  other  instruments  are  created  by  linking  the  assessment  to  important  individual  and 
group  goals  (139).  These  assessments  are  often  financially  expensive,  time  intensive,  and  are  not 
appropriate  for  the  constantly  changing  contexts  (physical,  social,  and  psychological)  where 
military  medical  leadership  exists.  In  comparison,  the  MMLA  has  no  financial  cost  and  is 
developed  with  uniformed  healthcare  leadership  in  multiple  contexts  as  a  primary  focus. 
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Increasingly,  the  uniformed  services  have  been  developing  their  own  leadership 
assessments  that  incorporate  a  “360  degree”  view-  that  is,  a  leadership  assessment  that 
incorporates  input  from  an  individual’s  superiors,  peers,  and  subordinates.  The  Department  of 
the  Army’s  Multisource  Assessment  and  Feedback  (MSAF)  is  the  most  widely  used  leadership 
assessment  instrument  within  the  Armed  Forces,  and  is  a  “360  degree”  assessment  instrument 
(10).  However,  the  “360”  approach  to  leadership  assessment  is  not  without  critics.  Togel  and 
Conger  (174)  note  that  “360”  assessments  often  include  both  objective  measures  of  performance 
and  more  qualitative  measures  of  leader  development.  For  the  leader  being  assessed,  this  mixed- 
subject  assessment  may  lead  to  confusion  and  frustration  from  feedback  that  may  be  more 
appropriately  separated  (i.e.,  a  leader  who  is  beloved  by  his  peers  and  subordinates  but  achieves 
few  organizational  goals;  a  leader  who  is  strongly  disliked  by  his  subordinates  but  admired  by 
peers  and  superiors;  a  leader  who  has  elicits  relatively  neutral  feelings  from  peers,  subordinates, 
and  superiors,  but  achieves  a  high  number  of  goals,  etc.).  Further  complicating  a  “360” 
assessment  is  that,  very  often,  all  individuals  within  an  organization  are  given  the  same 
assessment  to  complete,  regardless  of  time  in  the  organization,  seniority,  professional 
experience,  etc.  (74).  The  lack  of  specificity  can  lead  to  excessively  incongruent  feedback  that  is 
of  little  value  to  the  assessed  leader  and  may  actually  lead  to  disillusion  with  the  assessment 
process  and  decreased  leader  performance  (104).  Others  have  pointed  out  that  “360” 
assessments  may  be  limited  by  the  challenges  of  adapting  the  assessment  to  the  culture  of  the 
assessed  individual  and  the  organization  (114;  180).  Often,  “360”  assessments  are  labor- 
intensive  and  time  consuming,  which  may  limit  their  usefulness  in  the  fast-paced  environment  of 
uniformed  medicine  (182).  The  MMLA  is  especially  designed  for  USU  faculty  to  quickly  and 
easily  assess  uniformed  student  physicians  and  provide  clear  and  unambiguous  feedback  about 
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military  medical  leadership  performance.  Additional  uses  of  the  MMLA  as  a  self-  or  peer- 
assessment  is  discussed  further  in  Chapter  5. 

Current  Challenges 

The  MHS  has  identified  the  lack  of  a  meaningful  leadership  assessment  instrument 
as  a  critical  shortcoming  that  should  be  addressed  (103;  136).  If  this  shortage  is  to  be 
addressed,  then  any  leadership  assessment  instrument  must  be  sensitive  to  the  challenges  and 
critiques  presented  above.  Such  sensitivity  is  especially  important  to  military  medicine,  where 
the  culture  and  career  trajectory  of  healthcare  leaders  is  uniquely  different-  even  from  other 
leaders  within  the  same  unit.  Perhaps  nowhere  is  this  distinction  more  clear  than  within  the  Law 
of  War  (LOW),  a  collection  of  Federal,  Military,  and  International  Laws  that  proscribe  the 
conduct  of  modern  warfare  (127).  Nearly  every  uniformed  personnel  within  the  Department  of 
Defense  (including  the  Coast  Guard  during  times  of  conflict)  is  identified  as  a  “combatant”  who 
is  given  legal  sanction  to  engage  in  offensive  combat  between  two  or  more  recognized 
belligerent  states  (127).  Those  uniformed  personnel  who  fall  under  the  category  of  “non- 
combatant”  are  expressly  prohibited  from  waging  any  type  of  offensive  warfare  (127).  In  the 
uniformed  services,  such  a  classification  applies  only  to  chaplains  and  healthcare  personnel. 

This  separation  of  “provider  and  sustainer”  from  the  “warrior”  makes  sense  -  the  mission 
of  uniformed  medicine,  as  the  former  motto  of  the  Army  Medical  Department  succinctly  states, 
is  to  conserve  the  fighting  strength  of  American  warriors.  This  mission  is  so  expansive  and  all- 
encompassing  that  practically,  uniformed  healthcare  leaders  have  little  ability  to  join  their 
combatant  sisters  and  brothers  in  engaging  and  destroying  the  enemies  of  the  United  States. 
Ethically,  of  course,  the  principles  of  non-maleficence  are  so  central  to  medicine  in  the  United 
States  that  the  non-combatant  nature  of  uniformed  healthcare  is  a  widely  accepted  cultural  norm. 
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Additionally,  combatant  Warriors  often  have  a  wide  range  of  opportunities  to  lead  within  the 
DoD,  uniformed  medical  personnel  have  comparatively  fewer  options  to  develop  and  practice 
military  leadership  (103).  The  combination  of  inherent  non-maleficence  and  limited  leadership 
opportunities  available  to  develop  military  medical  leadership  may  be  why  leadership 
development  in  uniformed  healthcare  lags  behind  leadership  development  and  assessment  among 
combatant  Warriors.  By  developing  a  leadership  assessment  instrument  adapted  to  the  unique 
culture  of  military  medicine,  the  MMLA  seeks  to  narrow  this  gap. 

By  drawing  on  the  relative  strengths  of  TLT  augmented  by  the  FourCe  model  and 
engaging  leaders  from  diverse  backgrounds,  the  central  goal  of  this  project  was  to  create  a  brief 
leadership  assessment  tool  that  would  be  freely  accessible  to  faculty  and  educators  as  well  as 
sensitive  to  the  unique  requirements  of  developing  outstanding  uniformed  physician  leaders. 
These  various  leaders,  who  come  from  both  medical  and  non-medical  backgrounds,  collaborated 
using  a  structured  qualitative  research  method  called  the  Delphi  method,  described  in  detail  in 
Chapter  3.  This  project  represents  the  first  assessment  instrument  designed  specifically  to 
evaluate  uniformed  student  physicians  that  is  grounded  in  scientific  theory,  represents  a  joint 
Service  perspective,  and  incorporates  the  views  of  a  diverse  background  of  leaders  and  medical 
educators. 

Relevance  of  Present  Project 

The  creation  of  a  culturally-relevant  and  up-to-date  medical  leadership  assessment 
instrument  may  have  positive  impact  on  patient  care,  Warrior  fitness,  and  unit  readiness.  When 
examining  the  reasons  why  talented  and  capable  leaders  are  leaving  the  uniformed  services  for 
civilian  careers,  the  most  common  reason  that  veterans  (medical  and  non-medical)  give  is  not  the 
deployment  cycle  or  pay  but  rather,  toxic  leadership  (153;  169;  183).  A  meaningful  leadership 
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assessment  designed  for  military  healthcare  may  help  MHS  leaders  identify  toxic  leadership 
traits  in  their  subordinates  and  appropriately  correct  them.  Similarly,  an  assessment  instrument 
would  help  assist  uniformed  healthcare  leaders  to  identify  those  subordinates  who  demonstrate 
leadership  competence  for  more  senior  responsibilities  and  roles.  Additionally,  a  leadership 
assessment  instrument  focused  on  uniformed  medical  students  would  aid  leadership  development 
programs  and  policies  by: 

1)  Identifying  critical  elements  of  contemporary  military  medical  leadership 

2)  Standardizing  military  medical  leadership  development  and  definitions  in  a 
meaningful  way 

3)  Encouraging  future  scholarship  into  military  medical  leadership. 

In  early  2013,  COL  Charles  M.  Callahan,  M.D.,  then  commander  of  the  joint-Service  Ft. 
Belvoir  Community  Hospital  (FBCH),  edited  a  proposal  for  leadership  development  within  the 
AMEDD  that  he  and  his  fellow  contributors  call  “All  Physicians  Lead  (APL).”  This  ambitious 
program,  which  focuses  on  narrowing  warrior-provider  leadership  gap  described  above  through 
physician-led  small  group  discussions,  selected  readings  on  leadership  and  formal  leadership 
didactics  (132).  While  this  proposal  acted  as  a  call-to-action  for  physicians  to  take  the  initiative 
to  increase  resources  and  time  invested  into  medical  leadership  development,  one  of  the 
shortcomings  identified  by  APL  was  a  lack  of  a  medical  leadership  assessment  tool  (132).  The 
present  project  is  mindful  that  an  assessment  instrument  designed  for  uniformed  medical  students 
may  be  used  in  other  leadership  development  programs,  including  facilities  like  FBCH,  where 
USU  students  often  rotate  during  medical  training. 

Although  this  project  focuses  on  military  student  physicians,  there  is  applicability  to 
healthcare  leadership  outside  of  the  military  as  well.  The  military  provides  a  unique 
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environment  to  conduct  research  into  leadership  as  most,  if  not  all,  healthcare  providers  in  the 
military  are  leaders  because  of  their  rank,  position,  and  responsibilities.  Certainly,  all  of  the 
uniformed  medical  students  that  graduate  from  USU  are  to  be  military  medical  leaders  because 
USU  is  a  federal  service  academy  and  leadership  training  is  a  part  of  USU’s  mission.  The 
responsibilities  of  military  medicine  distinguish  the  field  from  civilian  healthcare.  Military 
medicine,  by  law  and  custom,  has  obligations  to  national  defense  (both  in  terms  of  physical 
security  and  public  health),  advancing  the  field  of  healthcare  (both  translational  and  clinical 
research),  and  when  directed,  fulfill  these  same  obligations  to  partner  and  allied  nations.  The 
examples  of  military  medicine  successfully  performing  these  tasks  are  numerous  and  are  briefly 
described  in  the  proposal  background.  On  an  even  deeper  level,  the  unique  culture  of  the 
military,  which  provided  much  of  the  most  influential  early  research  into  leadership,  warrants 
special  focus.  The  numerous  sub-cultures  that  exist  within  the  United  States  military,  as  well  as 
the  increasing  interagency  and  international  operations  of  military  medicine,  place  pressure  and 
demands  on  uniformed  healthcare  leaders  in  ways  previously  unknown  (142).  Additionally,  the 
military  continues  to  provide  in  situ  laboratories  for  emerging  healthcare  management  including 
patient  centered  medical  home  (PCMH)  (61).  Perhaps  the  most  important  reason  that  this  project 
focuses  on  military  medical  leaders  stems  from  the  increased  demand  to  provide  Warriors  and 
their  families  with  high  quality  care  from  uniformed  healthcare  providers  who  are  familiar  with 
the  demands  and  challenges  within  the  military  (126). 

Finally,  it  is  important  to  acknowledge  that  the  MMLA  has  clinical  implications  for 
healthcare  leadership  beyond  the  MHS.  Military  medicine  in  the  United  States  has  provided 
high  standards  of  clinical  practice,  medical  research,  and  numerous  healthcare  leadership 
examples  that  American  civilian  healthcare  has  emulated  for  generations.  If  history  serves  as 
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example,  then  development  and  assessment  of  outstanding  healthcare  leaders  at  USU  and 
throughout  the  MHS  may  strongly  influence  how  civilian  healthcare  leaders  are  developed  and 
assessed.  Respecting  the  influence  of  uniformed  healthcare  on  the  entire  nation  as  a  whole  is 
reflected  in  the  numerous  civilian  community  leaders  who  helped  to  create  the  MMLA.  The 
clinical  implications  of  the  MMLA  for  providers  and  patients  are  discussed  in  Chapter  5. 
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CHAPTER  2:  Research  Questions 

Framework 

This  project  used  qualitative  methodologies  to  design  a  new  assessment  instrument. 
Qualitative  methods  are  an  appropriate  research  approach  when  little  research  or  literature 
prevents  the  creation  of  meaningful  hypotheses  (35;  146;  164).  As  outlined  above,  little  is 
known  about  assessing  medical  leadership  -  still  even  less  is  known  about  assessment  of  military 
medical  leadership.  One  overarching  theme  of  the  present  project  is  to  provide  qualitative 
research  about  military  medical  leadership  assessment  aimed  at  providing  the  future  foundation 
for  testable  hypotheses. 

Although  qualitative  research  was  at  one  time  mischaracterized  as  excessively  anecdotal 
(9),  the  past  two  decades  of  research  using  qualitative  approaches  have  greatly  increased  in  both 
number  and  quality  (50).  Additionally,  qualitative  methods  such  as  the  Delphi  method  used  in 
this  project  are  especially  useful  for  research  into  healthcare  leadership  and  assessment,  which  is 
discussed  in  further  detail  in  Chapter  3.  The  present  project  directly  examined  leadership  in 
military  medicine  from  a  diverse  array  of  military,  community,  and  healthcare  leaders,  and  may 
be  considered  to  indirectly  demonstrate  that  qualitative  methods  are  relevant  in  military  and 
medical  populations. 

Research  Questions 

Unlike  traditional  quantitative  methods  which  present  specific  aims  and  linked 
hypotheses,  qualitative  designs  pose  research  questions  that  guide  and  inform  research  projects, 
while  still  maintaining  the  hallmark  flexibility  of  qualitative  research  (119).  The  questions 
themselves  are  largely  formed  from  the  constructivist  tradition  -  ascribing  meaning  to  multiple 
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perspectives  that  are  largely  open-ended  in  nature  with  the  intention  of  developing  a  meaningful 


pattern  (51).  The  following  research  questions  guided  this  project: 

1)  What  leadership  terms,  concepts,  or  ideas  do  leadership  experts  believe  are  important 
for  student  physicians  to  develop  at  USU?  [RQ1] 

2)  How  to  create  an  assessment  instrument  to  best  measure  the  leadership  terms, 
concepts,  or  ideas  identified  in  question  one?  [RQ2] 

3)  How  does  expert  consensus  relate  to  the  FourCe  Model  of  Leadership  and 
transformational  leadership  theory  (TLT)?  [RQ3] 

The  present  project  was  designed  to  answer  these  questions  by  collaborating  with  a  panel 
of  experts  in  leadership  development.  The  method  for  data  collection  used  in  this  project 
followed  the  Delphi  Method  which  is  described  in  the  following  chapter.  This  method  was 
selected  to  generate  an  unbiased  list  of  terms  and  concepts  that  were  refined  and  subsequently 
integrated  into  the  creation  of  a  new  military  medical  leadership  assessment  instrument 
(MMLA).  In  the  discussion,  the  new  MMLA  is  related  to  existing  theories  of  leadership, 
especially  Transformational  Leadership  Theory  (TLT)  and  the  FourCe  Model. 
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CHAPTER  3:  Methods 


Introduction  to  the  Delphi  Method 

The  Delphi  Method  is  named  after  the  Greek  oracle  that  provided  ancient  Hellenic 
military  leaders  with  guidance  and  counsel.  The  method  was  introduced  in  1950s  by  the  RAND 
corporation  for  a  USAF-sponsored  study  into  Soviet  military  strategy  (54).  Since  then,  the 
Delphi  method  has  been  used  to  create  novel  models  and  instruments  (53;  83;  165),  collect 
expert  consensus  (113;  157),  and  as  a  means  of  forecasting  and  predicting  outcomes  (157;  166). 

The  “classic”  Delphi  method  as  outlined  by  Rowe  and  White  (157)  has  four  essential 
features: 

1.  The  Delphi  method  participants  remain  anonymous  to  each  other  to  facilitate  frank 
opinions  without  pressures  to  social  conformity. 

2.  The  Delphi  method  consists  of  multiple  iterations,  or  “waves”  to  allow  participants  to 
refine  their  views. 

3.  The  Delphi  method  researcher  provides  “controlled  feedback”  -  communicating  the 
groups’  perspectives  while  maintaining  anonymity  of  the  group  members. 

4.  The  final  product  of  the  Delphi  method  is  an  aggregation  and  synthesis  of  the  group 
members’  responses. 

The  Delphi  method  can  broadly  be  described  as  a  structured  communication  technique  in 
qualitative  analysis  (150)  where  a  participant  panel  of  experts  communicates  with  solely  with  an 
investigator  while  at  the  same  time  collaborating  with  other  experts  in  the  same  field.  The 
method  has  a  number  of  important  advantages  but  perhaps  the  most  important  to  this  project  is 
that  the  method  allows  a  wide  variety  of  different  leadership  experts  from  across  the  country  and 
globe  to  collaborate  on  the  creation  of  an  assessment  tool,  without  complicated  logistics  and  the 
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pressures  to  conformity  present  in  other  methods  where  participants  meet  with  each  other  in 
person  (8;  157).  Additionally,  the  Delphi  technique  is  especially  useful  when  there  is  a  topic  of 
interest  that  may  represent  a  wide  range  of  backgrounds  and  experiences  (191).  The  diversity  of 
experiences  in  healthcare  leadership  may  be  considered  sufficiently  broad  and  varied  to  take 
advantage  of  the  Delphi  technique.  Finally,  the  Delphi  technique  is  useful  where  little  or  no 
research  exists  on  a  broad  topic  of  interest  (1 13).  This  method  is  an  appropriate  start  to  creating 
a  military  medical  leadership  assessment  (MMLA)  where  none  currently  exists. 

Contemporary  Delphi  Studies  in  Healthcare 

The  Delphi  method  is  a  popular  methodology  in  modern  healthcare  research  (56).  During 
the  past  decade,  the  Delphi  method  has  been  used  by  healthcare  educators  to  identify  critical 
domains  of  medical  competency  (47).  It  also  has  aided  healthcare  decision-makers  identify 
barriers  to  implementing  patient  safety  initiatives  (5).  Delphi  methods  have  assisted  healthcare 
providers  to  improve  diagnostic  criteria  for  challenging  chronic  medical  conditions,  including 
carpal  tunnel  syndrome  (77)  and  affective  disorders  (14).  Investigators  have  identified  research 
priorities  within  specific  medical  subfields  using  the  Delphi  approach  (12;  24;  123)  including 
patient-provider  communication,  health  disparities,  provider  well-being,  and  stress. 

Relevant  to  the  present  project,  the  Delphi  method  has  been  used  for  investigations  into 
health  care  leadership.  Carroll  (43)  used  a  Delphi  study  to  identify  important  skills  and  attributes 
of  nurse  executives  in  the  21st  century.  Calhoun  et  al.  (41)  developed  an  “interprofessional 
competency  model”  for  healthcare  leadership  by  using  Delphi  surveys  that  included  physicians 
(allopathic  and  osteopathic),  chiropractors,  podiatrists,  optometrists,  and  dentists. 

Additionally,  the  Delphi  method  has  been  used  for  leadership  research  within  the  MHS 
(117;  118;  162).  Studies  that  have  used  the  Delphi  method  in  the  MHS  have  identified 
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competencies  among  military  medical  administrators  (117;  162),  pharmacists  (125),  dentists 
(147),  and  nurses  (141).  These  studies  have  largely  focused  on  individual  competencies  related 
to  healthcare  management  and  have  not  focused  on  assessing  leadership  among  student 
physicians. 

A  systematic  review  of  80  Delphi  method  studies  by  Boulkedid  et  al.  (33)  indicated  that 
the  method  is  useful  if  the  expert  panel  is  appropriately  heterogeneous  and  motivated  to 
participate  for  the  entirety  of  the  investigation.  Keeney  (98)  concludes  that  although  the  Delphi 
method  is  largely  subjective  in  the  data  collected,  the  method  is  valuable  to  building  expert 
consensus  in  healthcare. 

It  is  important  to  note  that  throughout  the  Delphi  method  used  within  this  project,  the 
experts  remained  anonymous  to  each  other  and  that  communication  was  exclusively  limited  to 
the  individual  expert  and  the  researcher.  More  contemporary  Delphi  method  studies  may  have 
expert  participants  play  a  more  active  role  building  consensus  among  each  other  -  even  directly 
communicating  with  one  another  (91).  This  project  maintained  the  anonymity  of  the  individual 
experts  to  avoid  appealing  to  authority  and  pressures  to  conformity  (157)  which  may  be  a 
concern  when  conducting  studies  using  participants  from  varying  military  ranks  and  experience. 

In  broad  terms,  this  project’s  methods  included  the  following: 

I.  Identifying  and  recruiting  a  panel  of  leadership  experts 

II.  Generate  items  based  on  an  open  ended  query  regarding  assessing  medical  leadership  in 

uniformed  student  physicians 

III.  Generate  questionnaire  items  based  on  a  qualitative  analysis  of  expert  opinion  and  return 
these  items  to  the  panel  for  further  revision  and  refinement  (feedback). 
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IV.  Querying  the  expert  panel  about  how  to  best  categorize  the  items  in  the  draft 
questionnaire  based  on  the  FourCe  Model  (feedback). 

V.  Conducting  a  cognitive  interview  to  clarify  the  items  and  improve  usability. 
The  following  section  will  discuss  each  of  these  methods  in  greater  detail. 

Project  Methods 


Figure  1:  MMLA  Delphi  Process 

This  project  used  the  traditional  Delphi  method  described  by  Skulmoski  (166)  to  create  a 
new  military  medical  leadership  assessment  (MMLA)  instrument  to  assess  leadership  in 
uniformed  medical  students  by  faculty  at  USU. 

Project  Proposal  &  IRB  Approval:  This  project  was  approved  by  the  doctoral  project 
Committee  and  by  the  USU  Institutional  Review  Board  (IRB)  in  the  fall  of  2014.  A  copy  of  the 
IRB  approval  for  this  project  is  located  in  APPENDIX  D. 

Research  Sample:  This  phase  consists  of  identifying  the  Delphi  method  participants, 
commonly  referred  to  as  “the  experts.”  Skulmoski  (166)  asserts  that  there  are  four  essential 
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components  necessary  to  be  considered  an  expert:  (a)  knowledge  of  the  subject  matter  being 
studied,  (b)  willingness  and  ability  to  participate,  (c)  sufficient  time  to  participate,  and  (d) 
effective  communication  skills  (4). 

Working  with  leadership  experts  at  USU,  several  current  and  former  military  leaders 
(both  medical  and  line-Command)  from  each  military  service  (Army,  Navy,  Marine  Corps,  Air 
Force,  Coast  Guard)  who  met  the  criteria  of  leadership  expert  were  identified  and  asked  to 
participate  via  email  solicitation.  Representation  from  each  branch  of  service  is  unique  to  this 
project  and  gives  the  MMLA  a  joint-Service  perspective,  which,  as  noted  earlier,  is  increasingly 
becoming  the  norm  in  uniformed  healthcare.  The  panel  included  several  types  of  healthcare 
professionals  (physician  and  non-physician  doctoral  level  providers),  nurse  providers,  healthcare 
administrators,  and  non-commissioned  officers  (NCOs)  to  reflect  the  various  professional 
specialties  that  a  student  physician  is  being  trained  to  lead.  It  is  also  important  to  note  that  the 
expert  panel  included  a  greater  than  representational  (116)  number  of  ethnic  minorities  and 
women  because  gender  (66;  67)  and  ethnicity  may  influence  leadership  style  (7;  143;  178). 
Specific  demographic  data  about  the  panel  of  experts  in  Q1  is  presented  in  APPENDIX  F. 

Additionally,  several  ethnic  minority  civilian  leaders  (both  healthcare  and  community) 
were  also  identified  and  asked  to  serve  on  the  expert  panel  in  order  to  gain  a  broader  range  of 
views  on  leadership  and  address  the  potential  use  of  the  MMLA  in  non-military  settings, 
especially  within  civilian  communities  of  color  which  have  greater  access  to  healthcare  since  the 
passage  of  the  Affordable  Care  Act  (AC A). 

After  identifying  potential  experts,  each  individual  was  contacted  via  scripted  email  and 
asked  to  participate  on  the  panel  by  clicking  on  an  embedded  hyperlink.  The  link  brought  the 
participant  to  a  secure  website  hosted  by  Research.net.  Experts  were  asked  to  participate  in  at 
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least  three  “waves”  of  the  Delphi  method  during  the  consent  process  (see  APPENDIX  C  for  the 
study  information  sheet),  with  each  wave  lasting  approximately  30  days.  Participants  were  asked 
not  communicate  to  each  other  about  their  input  to  the  project.  Participants  received  information 
about  other  experts’  views  via  controlled  communication  from  the  researcher. 

Delphi  Q1  Design:  This  phase  consists  of  the  initial  Delphi  questionnaire  sent  to  the 
panel  of  experts.  Skulmoski  (74)  states  that  this  round  of  questions  should  be  broad  and 
relatively  open  ended.  Schmidt  (159)  refers  to  Q1  Design  as  the  “brainstorm”  period.  The 
Delphi  Q1  Design  has  parallels  with  Tuckman’s  (176)  “Storming”  phase,  where  team  members 
(albeit  in  the  case  of  Delphi  remain  anonymous  to  each  other)  will  spark  initial  creativity. 

The  Q1  questionnaire  asked  the  panel  member  to  provide  demographic  data  including 
gender,  race,  age,  branch  of  uniformed  service,  level  of  education,  and  to  briefly  describe  their 
own  leadership  experiences  to  present  day.  The  Q1  questionnaire  then  asked  the  participant  to 
generate  a  list  of  terms  and  concepts  related  to  assessing  medical  leadership  in  order  to  gain 
unbiased  feedback.  Experts  were  emailed  a  link  to  the  Q1  questionnaire  through  the  project’s 
secure  website  hosted  by  Research.net.  A  screenshot  of  the  Q1  Survey  is  in  APPENDIX  B. 

After  45  days  of  data  collection,  the  responses  were  collected  and  coded  by  three  independent 
evaluators. 

Delphi  Q1  Analysis:  In  the  traditional  Delphi  method,  raw  data  from  the  initial  survey  is 
collected  and  analyzed  using  the  qualitative  method  determined  by  the  researcher  (140).  After 
consulting  with  qualitative  analysis  experts  in  the  USU  Graduate  School  of  Nursing  and 
Department  of  Preventive  Medicine  and  Biometrics,  a  coding  strategy  was  developed  to  analyze 
the  raw  data  collected  from  the  Q1  Questionnaire.  Three  independent  coders  from  the  Grunberg 
research  lab  were  given  the  coding  strategy  (see  APPENDIX  G  for  a  copy  of  the  coding 
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instructions)  and  the  raw  data  collected  from  the  Q1  questionnaire  item  that  asked  the  expert 
panel  to  provide  terms,  concepts,  and  ideas  that  are  important  to  developing  and  accessing 
medical  leadership  in  uniformed  student  physicians.  The  coders  were  also  instructed  not  to 
communicate  with  each  other  about  how  they  coded  the  data.  After  one  week,  the  three  coders 
submitted  their  codes  to  the  lead  researcher  on  hardcopy  in  a  de-identified  envelope. 

Each  de-identified  list  of  codes  was  compared  to  each  other  for  identification  of  items 
to  be  used  for  the  Q2  survey.  The  lead  researcher  and  an  additional  reviewer  independently 
reviewed  each  list  of  codes.  During  this  independent  review,  the  reviewers  selected  codes  for 
retention  in  the  Q2  questionnaire  that  were  identical  among  all  three  coders.  The  reviewers  then 
compared  their  selections  and  retained  only  the  codes  that  were  identical  among  all  three  coders 
and  were  selected  by  both  reviewers.  A  total  of  62  items  were  identified  in  this  manner. 

Delphi  Q2  Design:  The  Q2  survey  included  a  list  of  70  terms  for  the  experts  to  rate 
importance  on  a  5-point  Likert  scale  (1  -  Not  Important,  2  -Slightly  Important,  3  -  Somewhat 
Important,  4  -  Moderately  Important,  5  -  Very  Important).  Of  the  70  terms,  62  were  terms 
originating  from  expert  input  in  Ql.  Eight  additional  terms  predicted  to  be  unrelated  to  relational 
theories  of  leadership  were  included  for  future  validity  studies  of  the  MMLA  and  included  such 
terms  as  “physical  attractiveness”  and  “popularity.”  A  screenshot  of  the  Q2  Survey  is  in 
APPENDIX  B.  The  survey  was  sent  out  to  the  experts  in  the  form  of  an  email  link  uniquely  tied 
to  the  email  address  provided  in  Ql  ensuring  the  integrity  of  the  expert  input. 

Delphi  Q2  Analysis:  After  approximately  30  days,  the  survey  was  closed  and  the  data 
were  analyzed.  Of  the  70  items  that  were  rated  by  the  experts,  53  items  were  rated  as  a  4.0  or 
higher,  which  was  set  as  the  threshold  rating  for  retention  in  the  first  draft  of  the  assessment 
instrument.  These  53  items  were  examined  by  the  project  coding  team  to  determine  if  the  items 
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were  conceptually  similar  enough  to  reduce  the  overall  number  of  moderately  important  terms  to 
a  total  more  suitable  for  the  draft  instrument.  Using  a  modified  Q-sort  and  working 
independently,  the  coding  team  reduced  the  number  of  53  items  to  25.  Additionally,  four  items 
from  the  validity  terms  list  were  selected  for  inclusion  in  the  first  draft. 

Delphi  Q3  Design:  The  purpose  of  the  Q3  survey  was  to  gain  expert  input  on  how  to 
classify  the  29  items  identified  for  inclusion  in  the  first  draft  of  the  MMLA  based  on  the  FourCe 
model  (80)  used  at  the  Uniformed  Services  University  (USU).  The  expert  panel  was  asked  to 
classify  each  of  the  29  items  into  one  of  the  four  categories  of  the  model:  Character, 
Communication,  Competency,  or  Context.  As  done  previously,  the  Q3  survey  was  sent  out  to 
the  experts  in  the  form  of  an  email  link  uniquely  tied  to  the  email  address  provided  in  Q1 
ensuring  the  integrity  of  the  expert  input.  A  screenshot  of  the  Q3  survey  is  in  APPENDIX  B. 

Q3  Survey  Analysis:  After  approximately  30  days,  the  survey  was  closed  and  the  results 
were  analyzed.  Items  were  assigned  to  a  category  based  on  the  weighted  average  closest  to  the 
corresponding  category  (i.e.,  1  -  Competence,  2  -  Communication,  3  -  Character,  4  -  Context). 
These  4  categories  formed  the  basis  of  the  first  draft  of  the  MMLA.  After  consulting  with 
leadership  research  experts  at  USU  and  the  Department  of  Defense,  the  first  draft  of  the  MMLA 
was  created.  Each  item  was  assessed  using  a  behavior-based  scale  in  order  to  minimize  biases 
inherent  to  qualitative-based  instruments.  The  items  were  then  placed  into  one  of  the  four 
categories  based  on  the  mode  of  the  expert  panel’s  response. 

Cognitive  Interview  1,  Lirst  Draft  of  MMLA:  Cognitive  interviewing  (Cl)  is  a  useful 
qualitative  research  tool  where  researchers  administer  draft  questions  to  a  group  of  potential 
users  while  collecting  verbal  information  about  the  draft  items.  Although  Cl  is  not  a  formal 
qualitative  research  interview,  the  method  allows  investigators  to  collect  valuable  information  to 
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determine  if  the  draft  survey  is  actually  generating  the  information  that  the  survey  author  wants 
(25).  Cl  was  included  in  this  project  to  enhance  face  and  content  validity,  and  to  gain 
stakeholder  input  without  the  complicated  logistics  of  more  traditional  focus  groups.  This 
project  followed  Willis’  (184)  simplified  approach  to  cognitive  interviewing. 

a)  Present  the  survey  question  in  its  original  form. 

b)  Collect  a  short  description  of  any  problems. 

c)  Correct  the  item  based  on  stakeholder  panel  member  input. 

For  this  proposal,  the  cognitive  interview  was  conducted  by  identifying  educators, 
leaders,  and  faculty  members  at  USU  who  would  be  in  a  position  to  use  the  draft  MMLA.  A  total 
of  22  individuals  were  identified  and  asked  to  participate  in  a  web-based  cognitive  interview  in  a 
manner  identical  to  the  Delphi  panel  experts.  These  cognitive  interviewees  were  sent  the  draft 
MMLA  via  an  email  link  unique  to  their  email  address.  For  each  of  the  25  questions  on  the  draft 
MMLA,  the  cognitive  interviewees  were  asked 

a)  Rate  the  clarity  of  the  item  on  a  scale  of  1  to  4. 

b)  Offer  any  brief  comments  to  improve  the  question. 

Detailed  information  about  the  demographics  of  the  cognitive  interviewees  is  presented 
in  Chapter  4.  A  screen  shot  of  the  cognitive  interview  is  located  in  APPENDIX  B. 

Cognitive  Interview  2,  Final  Draft  of  MMLA:  After  30  days,  the  cognitive  interview  was 
completed.  Based  on  critical  feedback  from  the  stakeholders,  the  lead  researcher  and  an 
additional  reviewer  made  minor  stylistic  adjustments  to  the  MMLA  as  well  as  add  criteria  to 
each  item  based  on  previous  literature  in  military  medical  leadership,  transformational  leadership 
theory,  and  psychosocial  theory.  An  additional  cognitive  interview  was  sent  out  to  the  cognitive 
interviewees  with  24  items  (one  item,  “Flexibility,”  was  dropped  as  the  interviewees  indicated 


51 


that  it  was  not  meaningfully  different  from  another  item,  “Adapting”).  After  30  days,  the 
second  cognitive  interview  was  concluded  and  the  interviewee  responses  were  collected  and 
analyzed.  Based  on  input  from  Cognitive  Interview  2,  minor  edits  were  made  to  make  the 
criteria  and  assessment  items  more  clear.  After  consulting  with  leadership  experts,  the  MMLA 
was  divided  into  two  components:  an  instruction  and  item  criteria  sheet,  and  an  assessment  grade 
sheet.  The  instructions  and  item  criteria  were  placed  on  a  document  generated  in  Microsoft 
Word.  The  score  sheet  was  generated  in  Microsoft  Excel  and  listed  each  of  the  24  items  and  the 
behavior-based  assessment  scale,  also  suggested  by  military  leadership  experts.  The  final  draft 
of  the  MMLA  instruction  sheet  and  score  sheet  is  located  in  APPENDIX  E. 

Product  Feasibility:  The  final  MMLA  instruction  sheet  and  score  sheet  were  then  taken  to 
several  leadership  evaluators  at  the  Uniformed  Services  University  and  the  United  States 
Military  Academy.  These  evaluators  were  given  the  MMLA  and  asked  to  imagine  an  individual 
whose  leadership  the  evaluator  is  knowledgeable.  The  evaluators  were  asked  to  rate  this  de- 
identified  individual  using  the  MMLA  and  offer  comments  about  the  amount  of  time  required  to 
complete  the  assessment,  as  well  as  how  well  the  MMLA  captured  this  unnamed  individual’s 
leadership  ability.  The  feedback  gleaned  from  these  leadership  evaluators  is  presented  in  Chapter 
4. 
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CHAPTER  4:  Results 


The  guiding  philosophy  of  the  project’s  data  analysis  is  to  measure  and  refine  expert 
panel  input  and  feedback  in  order  to  produce  a  practical  military  medical  leadership  assessment 
instrument  (MMLA).  This  project  is  designed  to  identify  important  components  of  developing 
medical  leadership  in  uniformed  medical  students  (Research  Question  1),  to  create  a  leadership 
assessment  instrument  based  on  those  important  components  (Research  Question  2),  and  to  relate 
expert  consensus  to  current  medical  leadership  models  used  at  USU  (Research  Question  3). 

Delphi  Panel  Demographics 

The  initial  panel  consisted  of  33  experts  which  exceeded  the  guideline  for  10-18  expert 
subjects  drawn  from  previous  investigations  (16;  47;  91;  138;  161).  Additionally,  oversampling 
the  initial  survey  allowed  the  project  to  tolerate  some  participant  dropout  from  the  subsequent 
Delphi  rounds.  Demographic  data  of  the  Q1  expert  panel  is  included  in  APPENDIX  F. 

Data  Analytic  Strategy 

Q1  Data  Analysis 

The  Q1  survey  asked  expert  panel  participants  to  list  as  many  terms,  concepts,  and  ideas 
about  military  medical  leadership  as  they  feel  appropriate.  A  total  of  33  experts  responded  to  the 
Q1  survey  which  was  closed  for  input  after  60  days.  The  open-ended  data  was  then  analyzed  and 
coded  into  items  by  five  different  members  of  the  research  team  who  did  not  discuss  their  coding 
strategy  with  each  other.  Only  items  that  were  coded  identically  among  the  five  research  team 
members  were  retained  for  the  Q2  Survey.  This  yielded  a  total  of  62  items.  An  additional  8 
items  predicted  to  be  unrelated  to  military  medical  leadership  were  added  to  allow  future 
validation  research  of  the  MMLA. 
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Q2  Data  Analysis 

The  second  survey  asked  the  expert  panel  to  rate  the  importance  of  the  generated  terms 
and  concepts  on  a  scale  of  1  (Not  Important),  2  (Slightly  Important),  3  (Somewhat  Important),  4 
(Moderately  Important),  and  5  (Very  Important).  The  selection  of  a  1-5  scale  is  based  on 
previous  investigations  using  the  Delphi  study  and  is  advantageous  in  that  it  provides  a  clear 
score  for  cutoff  criteria  -  greater  than  4.1,  or  “above  average”  (86;  91).  Expert  consensus  in  this 
project  used  Ulschak’s  (177)  criteria  of  80%  or  more  of  the  panel  rating  an  item  to  be 
“moderately  important.”  A  total  of  53  items  out  of  the  original  62  items  from  Q1  met  the 
threshold  criteria  to  be  considered  “above  average”  and  were  retained  in  the  final  Delphi 
interview.  After  reviewing  the  items,  the  coding  team  found  that  several  of  the  58  items  were 
closely  related.  The  coding  team  used  a  Q-sort  method  from  a  previous  leadership  assessment 
research  (158)  to  identify  which  items  were  closely  related  to  one  another  by  placing  each  item 
into  one  of  three  categories:  behaviors,  traits,  and  skills.  Among  a  group  of  two  or  more  similar 
items  within  each  of  the  three  categories,  the  item  with  the  highest  rating  was  selected  to 
represent  the  overall  concept  of  the  item  group  in  the  first  draft  of  the  MMLA.  Using  this 
method,  the  coding  team  reduced  the  total  number  of  items  from  53  to  25  distinct  items.  A  total 
of  28  of  the  original  33  experts  responded  to  the  Q2  survey.  The  items  and  their  rating  of 
importance  are  listed  below: 


Item 

Score 

Item 

Score 

Oral  Communication  Skills 

4.75 

Self-Awareness 

4.39 

Non-Verbal  Communication  Skills 

4.89 

Emotional  Intelligence 

4.36 

Written  Communication  Skills 

4.90 

Selflessness 

4.46 

Listening 

4.82 

Leading  by  Example 

4.86 
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Collaborating 

4.32 

Accountability 

4.64 

Motivating 

4.68 

Integrity 

4.93 

Mentoring 

4.36 

Patience 

4.29 

Problem  Solving 

4.61 

Adaptability 

4.50 

Decision  Making 

4.54 

Flexibility** 

4.39 

Time  Management 

4.43 

Confidence 

4.39 

Learning 

4.11 

Responsibility 

4.57 

Mission  Focus 

4.50 

Creativity 

4.21 

Cultural  Sensitivity 

4.25 

**Flexibility  was  not  included  in  the  final  draft  of  the 
MMLA  after  input  from  the  cognitive  interviewees. 

Figure  2:  MMLA  Items  &  Scores 

Q3  Survey  &  Analysis 

The  Q3  Survey  asked  the  panel  to  conduct  a  modified  Q-sort  of  the  items  that  met  the 
criteria  for  retention  as  described  above.  A  total  of  25  items,  along  with  4  items  that  were 
predicted  to  be  unrelated  to  military  medical  leadership  included  for  future  validity  studies  of  the 
MMLA.  The  Delphi  panel  was  asked  to  place  the  29  total  terms  into  one  of  four  categories  that 
consist  of  the  FourCe  Model  of  Medical  Leadership  used  at  USU:  Communication,  Competence, 
Character,  and  Context.  The  item  was  categorized  based  on  the  mode  of  the  panel’s  input.  A 
total  of  25  of  the  original  33  experts  responded  to  the  Q3  survey.  The  results  of  Q3  are  listed 
below,  with  the  total  number  of  the  25  experts  who  placed  an  item  into  one  of  the  FourCe 
categories. 
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FourCe 

Category 

Item 

Competence 

Communication 

Character 

Context 

Competence 

Decision  Making 

21 

1 

2 

1 

Problem  Solving 

19 

3 

0 

3 

Time  Management 

14 

2 

2 

7 

Learning 

14 

1 

2 

8 

Mission  Focus 

10 

2 

5 

8 

Communication 

Oral  Communication 

1 

23 

1 

0 

Written  Communication 

5 

20 

0 

0 

Non-Verbal  Communication 

0 

20 

2 

3 

Listening 

3 

19 

3 

0 

Collaborating 

5 

17 

2 

1 

Mentoring 

5 

14 

4 

2 

Character 

Integrity 

0 

0 

24 

1 

Patience 

1 

1 

22 

1 

Selflessness 

2 

0 

22 

1 

Responsibility 

4 

0 

20 

1 

Leading  by  Example 

3 

0 

18 

4 

Accountability 

3 

3 

15 

3 

Emotional  Intelligence 

3 

6 

13 

3 

Self-Awareness 

7 

1 

13 

4 

Confidence 

9 

1 

12 

3 

Adaptability 

7 

0 

12 

6 

Context 

Creativity 

7 

0 

7 

11 

Cultural  Sensitivity 

4 

4 

7 

10 

Figure  3:  MMLA  Items  in  the  FourCe  Model 

Cl  1  Survey  &  Analysis 

The  25  items  generated  and  rated  moderately  important  or  higher  was  then  fashioned  into 
the  first  draft  of  the  MMLA  instrument.  Additionally,  expert  input  from  Q3  determined  which 
categories  the  items  would  be  placed  within  the  MMLA.  As  described  above,  several  educators 
and  experts  within  USU  and  the  Walter  Reed  National  Military  Medical  Center  were  approached 
and  recruited  to  participate  in  a  cognitive  interview.  Each  question  on  the  first  draft  MMLA  was 
assessed  for  clarity  and  on  a  1-5  Likert  Scale:  Not  at  all  Clear  (Score=l),  Slightly  Clear 
(Score=2),  Mostly  Clear  (Score=3),  and  Very  Clear  (Score=4).  This  rating  was  adapted  from 
previous  surveys  using  the  cognitive  interview  method  (184).  Cognitive  Interviewees  were  also 
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asked  to  provide  brief  comments  to  improve  the  clarity  and  quality  of  the  item.  The  4  items 
predicted  to  be  unrelated  to  military  medical  leadership  were  not  included  in  the  cognitive 
interview. 

A  total  of  15  cognitive  interviewees  responded  to  the  Cl  survey  within  the  allotted  30 
days,  meeting  the  suggested  requirement  of  10-15  interviewees  established  by  previous  studies 
(185).  Based  on  the  ratings  of  clarity  and  the  comments  provided  by  the  interviewees,  24  items 
were  retained  for  an  additional  cognitive  interview.  One  item,  “Flexibility”  was  critiqued  by 
several  of  the  interviewees  as  being  too  similar  to  “Adaptability.”  For  the  second  cognitive 
interview,  “Adaptability”  was  retained  bringing  a  total  of  24  items  to  the  final  draft  of  the 
MMLA.  Additionally,  a  common  critique  among  the  panel  was  the  lack  of  rating  scale  and  item 
criteria.  For  the  second  cognitive  interview,  criteria  for  each  item  was  developed  based  on 
concepts  developed  from  Transformational  Leadership  Theory  (TLT)  (20),  the  USU  FourCe 
Leadership  Model  (80),  and  Psychosocial  Theory.  Additionally,  a  behavior-based  rating  scale 
was  developed  after  consulting  with  military  behavioral- science  experts  who  indicated  that  such 
a  scale  would  minimize  bias  and  allow  for  a  more  objective  and  specific  leadership  assessment 
(120).  The  scale  developed  for  the  MMLA  ranged  from  Almost  Never — Infrequently — 
Sometimes — Often — Almost  Always. 

C2  Survey  &  Analysis 

A  second  cognitive  interview  was  sent  to  the  interviewees  and  asked  them  to  rate  the  item 
criteria  and  assessment  question  for  each  of  the  24  items,  as  well  as  offer  any  comments  to  make 
the  items  more  clear.  A  total  of  10  interviewees  responded  to  the  cognitive  interview  in  the  30 
days  allotted  for  the  C2  survey,  meeting  the  requirement  of  10-15  interviewees  suggested  by 
previous  research  (184;  185).  A  total  of  8  items  that  were  rated  lower  than  3  (Moderately  Clear) 
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were  examined  and  adjusted  based  on  the  comments  provided  by  the  interviewees.  The  final 
draft  of  the  MMLA  is  included  in  APPENDIX  E. 

Product  Feasibility 

The  MMLA  was  then  brought  to  a  small  group  of  leadership  evaluators  at  the  United 
States  Military  Academy.  The  evaluators  were  asked  to  assess  an  anonymous  individual  whom 
the  evaluator  is  familiar,  especially  in  terms  of  leadership  style,  skills,  and  behaviors.  The 
evaluator  was  then  asked  how  long  the  assessment  of  the  anonymous  individual  took,  how  well 
the  assessment  captured  the  anonymous  individual’s  leadership  behaviors,  and  if  the  assessment 
would  be  useful  in  potentially  assisting  the  leadership  development  of  the  anonymous  individual. 
A  total  of  4  individuals  completed  this  exercise.  The  average  time  to  complete  the  assessment 
was  approximately  3  minutes. 

Comments  from  USMA  Evaluator  1:  After  completing  the  MMLA,  Evaluator  1 
mentioned  that  as  he  was  assessing  the  anonymous  individual,  he  was  simultaneously  assessing 
his  own  leadership  abilities  among  the  24  items.  He  stated  that  this  was  a  unique  quality  of  the 
MMLA  and  that  he  had  never  simultaneously  self-assessed  while  evaluating  another  individual. 
Evaluator  1  suggested  that  the  MMLA  be  used  as  a  self-assessment  measure  as  well  as  a  faculty- 
to-student  assessment. 

Comments  from  USMA  Evaluator  2:  Evaluator  2  stated  that  he  was  pleased  with  the  use 
of  emotional  intelligence  as  an  item  related  to  leadership.  Evaluator  2  also  stated  that  the 
concepts  of  emotional  intelligence  and  self-awareness,  while  familiar  to  younger  individuals, 
may  not  be  as  widely  understood  by  older  evaluators.  He  suggested  that  any  evaluator  using  the 
MMLA  be  familiar  with  the  concepts  of  emotional  intelligence  and  self-awareness,  and  that 
these  two  items  may  not  necessarily  be  mutually  exclusive. 
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Comments  from  USMA  Evaluator  3:  Evaluator  3  took  approximately  two  minutes  to 
complete  the  assessment.  He  then  compared  it  to  the  current  assessment  tool  used  at  USMA  and 
suggested  that  the  MMLA  be  digitized  for  faculty  portability.  Evaluator  3  also  noted  that  the 
behavior-based  scale  was  simpler  to  assess  than  qualitative  scales  often  used  in  military 
leadership  assessments. 

Comments  from  USMA  Evaluator  4:  Evaluator  4  took  the  longest  time  of  the  four 
evaluators,  completing  the  MMLA  in  approximately  four  minutes.  She  was  interested  in  how  the 
items  were  generated  and  suggested  that  the  instruction  sheet  might  include  a  brief  statement  as 
to  where  the  items  came  from  in  order  to  give  the  MMLA  source  credibility  with  uniformed 
evaluators. 
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CHAPTER  5:  Discussion 


Leadership  in  uniformed  medicine  has  been  identified  as  a  critical  need  by  the 
Department  of  Defense,  the  individual  uniformed  services,  and  the  Uniformed  Services 
University.  The  purpose  of  this  project  was  to  create  a  brief  military  medical  leadership 
assessment  (MMLA)  instrument  through  the  Delphi  Method,  a  structured  qualitative 
communication  technique  where  experts  work  collaboratively  while  remaining  anonymous  to 
each  other.  The  Delphi  Method  consists  of  several  rounds  of  survey  interviews  between  the 
expert  panel  and  the  investigator,  who  analyzes  the  data  and  provides  controlled  feedback  to  the 
panel.  In  this  project,  33  leadership  experts  volunteered  to  collaborate  in  order  to  generate  items 
thought  to  be  important  to  assessing  uniformed  student  physicians  in  medical  leadership. 
Subsequent  coding  from  the  project  research  team  identified  25  unique  items.  An  additional 
interview  method,  cognitive  interviewing,  improved  the  mechanics  and  clarity  of  the  items  into  a 
brief  leadership  assessment  instrument  containing  24  items  categorized  into  the  four  domains  of 
the  FourCe  Medical  Leadership  Model  and  linked  to  a  behavior-based  rating  scale. 

This  project  had  three  research  questions  which  will  frame  the  discussion  throughout  this 
chapter.  For  each  research  question,  the  projects  results,  limitations,  and  future  directions  will  be 
presented.  At  the  conclusion  of  this  chapter,  the  overall  clinical  implications  of  the  Military 
Medical  Leadership  Assessment  (MMLA)  will  be  discussed. 
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Research  Question  1  (RQ1) 

What  leadership  terms,  concepts,  or  ideas  do  leadership  experts  believe  are  important  for  student 
physicians  to  develop  at  USU? 

RQ1  Results 

To  answer  and  discuss  RQ1,  the  first  two  rounds  of  Delphi  Panel  surveys  offer  the  most 
interpretation.  Several  hundred  words  and  phrases  were  generated  by  the  initial  33  leadership 
experts  in  the  first  round  of  Delphi  method  surveys.  The  purpose  of  this  stage  of  the  project  was 
to  generate  unprompted  and  unbiased  feedback  in  order  to  build  the  foundation  for  an  assessment 
instrument  where  one  currently  did  not  exist.  As  with  most  qualitative  research,  some  level  of 
coding  analysis  must  be  performed  in  order  to  make  the  relatively  large  amount  of  unfiltered  and 
open-ended  data  into  useful  information  required  to  answer  RQ1.  This  project  took  5  individual 
coders  that  independently  examined  the  data  generated  by  the  33  Delphi  experts  and  coded  them 
into  62  terms  that  each  coder  unanimously  agreed.  Although  62  items  were  identified,  these 
items  had  yet  to  be  identified  as  important  by  expert  consensus.  An  additional  Delphi  survey 
round  would  be  required  in  order  for  the  panel  to  identify  which  of  the  62  items  were  sufficiently 
important  to  assessing  leadership  in  uniformed  medical  students.  The  second  Delphi  Panel 
survey  (Q2)  and  subsequent  Q-sort  from  the  project  coding  team  identified  25  items  as  unique 
and  meeting  the  previously  set  threshold  of  S>4.0. 

According  to  the  expert  panel,  items  rated  very  important  included  integrity,  leading  by 
example,  and  effective  communications  skills.  The  importance  ascribed  to  these  items  indicate 
that  leadership  experts  desire  to  see  student  physicians  with  a  strong  internally  consistent  moral 
code,  as  well  as  the  ability  to  effectively  express  information  in  multiple  ways.  These  findings 
correspond  well  with  this  project’s  focus  on  developing  leadership  assessment  using  relational 
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leadership  theories  -  Transformational  Leadership  Theory  (TLT)  in  particular.  Returning  to  the 
four  major  components  of  TLT:  Idealized  Influence,  Motivation,  Individualized  Consideration, 
and  Intellectual  Stimulation  (19),  the  items  identified  by  the  expert  panel  are  consistent  with 
TLT.  Leading  by  example  can  be  linked  to  Idealized  Influence  as  TLT  asserts  that  a 
Transformational  Leader  must  provide  an  appropriate  positive  role  model.  The  MMLA  items  of 
motivation,  leading  by  example,  and  integrity  may  address  a  criticism  of  TLT:  the  emphasis  on 
charisma.  The  focus  on  charisma  in  TLT  inspires  Khoo  (101)  and  others  to  warn  against  the 
“dark-side”  of  charisma  using  examples  of  charismatic,  yet  nefarious  leaders  throughout  history. 
A  leader’s  integrity  -  an  internally  consistent  and  strong  set  of  moral  principles  -  may  help  to 
address  this  particular  criticism  of  TLT.  Additional  discussion  about  the  MMLA  and  current 
leadership  is  presented  later  in  this  chapter  under  research  question  3  (RQ3). 

To  summarize,  this  project  was  among  the  first  leadership  assessments  that  has  focused 
on  medical  leadership  in  uniformed  student  physicians  that  brought  various  leaders  from  all 
branches  of  the  Department  of  Defense  (DoD)  and  the  civilian  community  to  collaboratively 
identify  terms,  ideas,  and  concepts  important  to  military  medical  leadership.  In  answering  RQ1, 
a  diverse  group  of  Leadership  experts,  medical  school  educators,  and  qualitative  coders 
identified  24  total  items  assessed  to  be  moderately-to-very  important  that  are  distinct  and  can  be 
subsequently  placed  into  a  draft  assessment  instrument. 

RQ1  Limitations 

The  project’s  limitations  in  answering  RQ1  include  the  coding  strategy  used  to  identify 
the  MMLA  items  from  the  raw  and  unbiased  data  collected  in  the  first  round  of  Delphi 
interviews.  The  coding  strategy  used  in  this  project,  while  reasonably  following  methods  used  in 
other  qualitative  projects  (17),  such  as  using  multiple  independent  raters  and  purposeful 


62 


sampling,  lacks  an  important  element  in  qualitative  research:  grounded  theory.  As  explained  in 
the  project’s  background,  the  purpose  of  this  project  was  to  identify  terms  and  concepts 
important  to  developing  leadership  in  uniformed  medical  students  and  not  to  advance  a  new 
theory  of  military  medical  leadership.  Although  this  project  proposes  some  new  definitions  and 
draws  from  established  leadership  theories,  further  research  is  needed  to  develop  a  new  and 
culturally  competent  grounded  theory  of  military  medical  leadership.  Some  future  directions  for 
developing  models  of  military  leadership  are  expounded  below. 

RQ1  Future  Directions 

These  24  items  are  perhaps  one  of  the  best  foundations  for  the  creation  of  a  new 
grounded  theory  of  military  medical  leadership  that  may  help  to  address  the  major  limitation  of 
coding  strategy.  In  the  most  direct  terms,  new  grounded  theories  are  generated  by  the  data 
collected  (75).  The  project’s  24  items  should  be  examined,  critiqued,  and  re-evaluated  in  real- 
life  settings  (e.g.,  using  the  MMLA  with  uniformed  student  physicians)  in  order  to  A)  validate 
the  new  FourCe  Leadership  model  proposed  at  USU  as  a  definitive  grounded  theory  of  military 
medical  leadership,  and  B)  identify  additional  components  of  military  medical  leadership  theory 
that  is  not  currently  reflected  in  the  FourCe  Model. 

Additionally,  these  items  should  not  be  viewed  as  static  or  fixed.  Indeed,  a  common 
criticism  has  been  that  the  recent  field  of  leadership  research  has  been  too  constrained  and 
inflexible  to  natural  changes  in  evolving  social  and  psychological  environments  (72).  As  these 
items  form  the  foundation  for  potential  advancements  in  grounded  theory,  this  project’s  items 
may  adjust  to  be  more  representative  of  advancements  in  both  theory  and  application  in 
specificity  or  sensitivity.  Such  adjustments  might  be  made  by  asking  leadership  experts  in  other 
fields  of  healthcare  (i.e.,  what  items  are  important  for  student  nurses  to  develop)  or  in  entirely 
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different  professions  including  among  military  leaders  in  training,  where  uniformed  doctrine 
often  forms  the  basis  of  assessment.  It  would  be  useful  for  future  studies  to  continue  to  develop 
the  items  in  order  to  identify  what  (if  any)  items  are  unique  to  healthcare  leadership  and  what 
items  may  transcend  organizational  and  professional  cultures. 

Research  Question  2  (RQ2) 

How  to  create  an  assessment  instrument  to  best  measure  the  leadership  terms,  concepts,  or 
ideas  identified  in  question  one? 

RQ2  Results 

Addressing  RQ2  focuses  the  discussion  on  the  project  coding  strategy,  and  the  cognitive 
interview.  These  stages  of  the  project  composed  the  bulk  of  the  creation  of  the  Military  Medical 
Leadership  Assessment  (MMLA). 

Prior  to  the  recruitment  of  the  first  Delphi  expert  panel  member,  the  author  informally 
surveyed  several  leadership  experts  at  the  Uniformed  Services  University  in  order  to  gain  a  sense 
of  what  a  useful  assessment  instrument  might  physically  resemble.  These  experts  reported  that 
an  ideal  assessment  would  be  relatively  brief  and  able  to  support  the  assessment  of  several 
student  physicians  over  multiple  time  points  throughout  their  medical  school  training.  Such 
comments  were  similarly  reflected  within  the  cognitive  interview,  where  interviewees  often 
critiqued  the  assessment  as  being  too  vague  and  lengthy  to  be  practically  useful. 

It  was  in  this  spirit,  that  the  MMLA  was  purposefully  crafted  to  be  portable,  limited  in 
length,  and  as  specific  as  practically  possible  -  which  is  reflected  in  the  coding  strategy  used  by 
the  project  coding  team.  The  process  of  fashioning  the  assessment  instrument,  although 
projective,  highlights  the  strength  of  the  Delphi  method.  From  all  over  the  world,  a  highly 
diverse  and  talented  group  of  leadership  experts  were  able  to  collaborate  on  the  creation  of  a  new 
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medical  leadership  assessment.  In  the  first  round  of  surveys,  a  large  collection  of  raw  data  was 
collected  that  yielded  in  excess  of  100  distinct  codes  among  6  individual  coders  working 
independently.  Although  these  codes  were  interesting,  an  additional  refinement  in  the  form  of 
two  additional  Delphi  surveys  and  a  Q  sort  from  the  project  coding  team  were  necessary  to  pare 
the  items  down  to  a  number  that  would  support  the  project’s  underlying  ethos  of  being  physically 
practical  to  the  intended  end  user:  leadership  assessors  at  the  Uniformed  Services  University. 

However,  without  a  grounded  theory  and  large  coding  team,  an  additional  component  of 
instrument  creation  would  be  necessary.  The  project’s  cognitive  interview  partially  addressed 
the  project’s  methodological  limitation  by  engaging  the  cognitive  processes  of  individuals  who 
may  largely  be  characterized  as  potential  users  of  the  new  instrument.  Cognitive  processes 
especially  useful  in  this  project  included  overall  attention,  comprehension  of  the  items,  and 
memory  -  elements  that  are  critical  to  developing  content  validity  in  new  healthcare  evaluation 
instruments  (144).  The  project’s  cognitive  interviews  have  provided  some  foundational  structure 
that  enables  the  intended  faculty  evaluator  increased  clarity  and  ease  while  the  MMLA 
undergoes  formal  validation. 

The  end  result  of  research  question  2  (RQ2)  saw  the  creation  of  a  three  page  assessment 
instrument  to  assess  medical  leadership  in  student  physicians.  Based  on  critical  feedback  from 
both  rounds  of  the  cognitive  interview,  the  items  were  described  with  brief  criteria  developed 
from  established  definitions  and  theories  including  Lewinian  leadership  psychology, 
psychosocial  development,  and  Transformational  Leadership  Theory  (TLT).  A  new  behavior- 
based  scale  was  placed  on  a  separate  single-page  score  sheet  that  also  included  a  space  for  free- 
text  comments  tied  to  each  item  as  well  as  overall.  This  is  the  first  leadership  assessment 
instrument  that  has  incorporated  a  purposefully  diverse,  joint- service,  and  inter-professional 
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perspective  on  developing  leadership  in  student  physicians.  The  strength  of  this  instrument  lies 
in  the  fact  that  it  was  created  with  the  conceptual  and  technical  input  of  the  leaders  and  educators 
that  would  be  most  likely  to  use  the  MMLA  in  various  settings. 

RQ2  Limitations 

The  most  important  limitation  of  the  project  in  addressing  RQ2  is  that  the  MMLA,  while 
created  using  sound  methods  and  from  accepted  theories,  remains  an  instrument  that  requires 
validity.  Validation  of  the  MMLA  will  likely  take  additional  research  in  order  to  establish  if  the 
instrument  is  meaningfully  measuring  an  individual’s  current  leadership  prowess,  as  well  as 
longitudinally  measure  performance  over  time. 

Another  limitation  of  the  MMLA  when  answering  RQ2  is  the  subjective  input  from  the 
project’s  coders.  For  this  project,  three  doctoral  students,  and  two  research  associates  well 
versed  in  psychosocial  theory,  transformational  leadership,  and  research  methods  were  utilized  to 
analyze  and  codify  the  data  at  two  critical  time  points.  Although  these  individuals  were 
knowledgeable  and  used  strategies  from  previous  leadership  assessment  projects,  additional 
coders  with  more  extensive  experience  in  qualitative  analysis  may  have  yielded  more  sensitive 
and  specific  items.  As  the  same  coders  were  used  throughout  the  project,  the  reliability  of  the 
MMLA  may  be  limited. 

Similarly,  while  the  cognitive  interviews  conducted  for  this  project  were  extremely 
useful,  they  were  limited  to  approximately  10  interviewees  that  communicated  through  an 
online -based  survey.  Other  forms  of  cognitive  interviewing  may  take  place  using  in-person 
focus  groups  where  the  researcher  is  able  to  gather  larger  amounts  of  verbal  and  non-verbal  data 
related  to  an  instrument’s  clarity,  utility,  and  ease  of  use.  In  person  cognitive  interviews  with 
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larger  groups  of  MMLA  users  may  help  to  address  this  limitation  as  the  measure  undergoes 
validation. 

RQ2  Future  Directions 

The  next  essential  step  for  the  MMLA  is  the  use  of  the  instrument  to  assess  leadership 
among  student  physicians  at  the  Uniformed  Services  University  (USU)  and  elsewhere. 
Fortunately,  the  MMLA  is  (at  the  time  of  this  publication)  being  considered  for  use  in  the  2015 
cycle  of  Operation  Bushmaster  and  among  student  physicians  at  the  University  of  South  Florida 
(USF).  With  the  valuable  input  of  the  Delphi  panel  and  the  cognitive  interviewees,  the 
instrument  may  be  reasonably  assumed  to  have  adequate  face  and  content  validity  (40). 

What  remains  necessary  is  for  the  MMLA  to  undergo  rigorous  formal  criterion  and 
construct  validity,  as  well  as  analysis  of  the  reliability  of  the  MMLA  among  a  wide  range  of 
assessors,  leaders,  and  educators.  Such  validation  should  be  initiated  not  only  at  the  Uniformed 
Services  University,  but  by  the  broad  variety  of  uniformed  and  civilian  institutions  represented  in 
the  Delphi  panel.  Future  validation  of  the  MMLA  may  also  include  factor  analysis  of  the  items 
as  related  to  each  other,  which  may  address  a  frequent  criticism  of  leadership  assessment, 
namely,  that  items  are  too  interrelated  (172). 

Proceeding  forward,  the  MMLA  should  be  used  in  organizations  where  effective  (and 
even  ineffective)  leaders  have  already  been  identified.  Leadership  evaluators  in  these 
organizations  should  evaluate  these  known  “leadership  quantities”  using  the  MMLA  in  order  to 
evaluate  the  instrument’s  sensitivity  (i.e.,  the  efficacy  of  the  MMLA  to  distinguish  between 
effective  and  ineffective  leaders).  As  the  MMLA  reflects  the  trend  within  leadership  assessment 
to  focus  on  behaviors,  a  succinct  way  of  testing  the  instrument’s  sensitivity  is  to  assess  medical 
leaders  who  have  already  excelled  in  challenging  leadership  positions.  Additionally,  the 
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MMLA’s  external  validity  should  be  determined  by  comparing  the  instrument  with  other 
commonly  used  leadership  assessments,  including  the  Army’s  Multisource  Assessment  and 
Feedback  (MSAF)  and  the  validated  Multifactor  Leadership  Questionnaire  (MLQ)  described  in 
Chapter  one.  The  MMLA  also  should  be  compared  among  leadership  assessment  tools  used  at 
the  other  Federal  Service  Academies  where  leadership  assessment  is  commonly  linked  to 
service- specific  military  doctrine  (131).  Use  of  the  MMLA  in  the  Federal  Service  Academy 
system  may  help  leadership  development  faculty  at  all  six  institutions  to  identity  and  address 
gaps  with  the  leadership  training.  If  the  instrument  is  sensitive,  valid,  and  reliable  to  accurately 
assess  and  predict  success  as  a  medical  leader,  then  the  MMLA  also  may  be  a  teaching  tool  to 
help  students  and  faculty  develop  into  more  effective  medical  leaders.  Once  psychometric 
properties  are  sound,  the  MMLA  might  be  used  to  compare  students  trained  at  USU  versus 
Health  Professions  Scholarship  Program  (HPSP)  students  -  the  other  major  source  of  physicians 
for  the  military. 

Research  Question  3  (RQ3) 

How  does  expert  consensus  relate  to  the  FourCe  Model  of  Leadership  and  transformational 
leadership  theory  (TLT)? 

RQ3  Results 

Discussion  of  the  third  research  question  will  center  on  both  the  final  Delphi  Q3  survey, 
as  well  as  the  content  generated  by  the  experts  with  contrast  to  the  FourCe  Model  and  TLT.  To 
frame  the  discussion  in  reference  to  TLT,  a  visual  representation  and  brief  description  of  the 
theory  is  presented  below  (20) 
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Figure  4:  Transformational  Leadership  Model 
Judge  (96)  offers  a  brief  and  useful  synopsis  of  each  domain  of  TLT: 

•  Individualized  Consideration:  Attending  and  supporting  the  needs  of  followers. 

•  Idealized  Influence:  Being  a  charismatic  role  model. 

•  Intellectual  Stimulation:  Stimulating  creativity  and  challenging  the  status  quo. 

•  Inspirational  Motivation:  Appealing  to  the  motivation  of  followers  by  providing  a  clear 
vision. 

A  brief  glance  of  the  domains  demonstrates  a  number  of  items  included  in  the  MMLA. 
Among  these  are  mentoring  others,  motivating  others,  creativity,  selflessness,  and  leading  by 
example.  In  contrasting  with  the  FourCe  model,  the  greatest  number  of  items  were  matched  with 
the  Character  category.  The  number  of  items  in  the  Character  category  highlights  how  the 
FourCe  model  and  the  MMLA  is  building  on  the  Transformational  Leadership  Model, 
addressing  the  critique  highlighted  earlier  about  charismatic-yet-toxic  leaders.  Additionally,  one 
of  the  most  important  strengths  of  the  FourCe  Model  and  the  MMLA  is  the  inclusion  of  Context, 
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which  is  critical  in  military  medicine  and  is  reflected  by  the  definition  of  military  medical 
leadership  also  presented  in  Chapter  1.  The  definition  is  presented  below  for  reference: 

Military  Medical  Leadership:  The  enhancement  of  behaviors,  cognitions,  and 
emotions/motivations  to  achieve  goals  that  benefit  individuals  and  groups  within  the  military 
healthcare  system  (MHS),  foster  the  health  of  Warriors  and  their  families,  and  supports  the  line 
Command  mission. 

Arguably,  each  of  the  24  items  presented  in  the  MMLA  could  potentially  be  found  within 
the  definition  of  military  medical  leadership  asserted  in  this  project.  In  terms  of  the  variety  of 
contexts  found  within  military  medicine,  the  MMLA  is  distinct.  The  responsibility  to  support  the 
Command  mission,  a  facet  unique  among  uniformed  physicians,  is  reflected  and  assessed  in  the 
MMLA  more  directly  than  other  leadership  assessment  instruments.  While  not  a  primary  aim  of 
this  project,  the  MMLA  has  the  potential  to  develop  transformational  leadership  theories  and 
definitions  in  an  uncertain  future  for  uniformed  and  civilian  medicine. 

Additionally,  a  component  of  the  definition  presented  above  and  the  MMLA  is 
motivation.  The  MMLA  measures  “motivating  others”  directly,  and  measures  motivation 
indirectly  through  “mission  focus,”  or  how  a  leader  manages  to  motivate  subordinates  and  peers 
to  remain  motivated  to  accomplish  difficult  or  challenging  mission. 

RQ3  Limitations 

As  stated  above  in  the  discussion  of  RQ2  limitations,  the  principle  limitation  of  this 
project  is  that  it  is  tied  to  the  FourCe  model,  which  continues  to  be  developed.  There  are  strong 
links  to  the  more  established  Transformation  Leadership  Model  (TLM),  however,  this  is  only 
one  model  of  many.  Furthermore,  leadership  theories  (including  TLM)  continue  to  be  critiqued 
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as  lacking  sufficient  scholarship  and  empirical  evidence  (55).  Even  the  process  of  developing 
grounded  theories  of  leadership  are  criticized  as  focusing  too  much  on  the  processes  between 
leader  and  follower,  and  insufficiently  on  developing  leadership  as  a  distinct  and  independent 
natural  phenomenon  (99).  The  MMLA,  although  possessing  face  and  content  validity,  is  limited 
by  the  relatively  short  history  of  scholarship  in  leadership  development  and  assessment  when 
compared  to  the  centuries  of  anecdotal  leadership  commentary  and  belief. 

RQ3  Future  Directions 

The  MMLA  will  not  settle  the  important  debates  of  how  leadership  theory  is  developed 
and  evaluated.  However,  the  MMLA  has  the  potential  to  address  an  important  critique  from  Day 
and  colleagues  (55),  who  argued  that  insufficient  attention  has  been  given  to  leadership 
development  in  favor  of  literature  that  is  scant  on  details  of  developing  effective  leaders.  The 
MMLA  also  puts  specific  details  into  the  FourCe  Model  of  Medical  Leadership.  While  future 
validation  may  shift  the  24  items  from  their  current  category  to  another,  the  MMLA  enables  the 
FourCe  Model  to  distinguish  itself  from  other  leadership  models  that  are  vague  on  specific 
details.  As  the  MMLA  is  used  to  assess  and  develop  individual  student  physicians  and  provide  a 
metric  of  the  efficacy  of  leadership  education  at  USU,  so  too  should  the  instrument  be  used  to 
provide  valuable  grounded  evidence  to  develop  existing  theories  of  leadership. 

This  project  was  the  first  step  in  developing  the  MMLA  which  needs  to  be  validated  to 
assess  medical  leadership.  In  addition  to  the  next  steps  described  above,  the  next  steps  for  USU 
may  include  using  the  MMLA  in  comparison  with  clerkship  evaluations,  STEP-1  and  STEP-2 
scores,  class  GPA,  residency  matches,  and  other  more  academic  assessments  widely  in  use. 
Similar  to  currently  serving  medical  leaders  who  are  of  a  “known  quantity”  in  terms  of  their 
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leadership  performance,  the  MMLA  should  be  used  to  assess  and  predict  those  students  who  take 
challenging  roles  and  assignments  post-graduation. 

Implications  for  Clinical  Practice 

One  of  the  major  drives  for  the  creation  of  the  MMLA  was  and  remains  improving  the 
quality  of  healthcare  provided  to  Warriors  and  their  families.  The  most  direct  means  that  the 
MMLA  accomplishes  this  is  enabling  experienced  medical  educators  and  leaders  to  assess  the 
leadership  ability  of  student  physicians.  The  link  between  improved  patient  outcomes,  patient 
safety,  and  effective  leadership  (especially  Transformational  Leadership)  has  been  observed  in 
multiple  studies  and  settings  (122;  187;  188).  Effective  leadership  has  also  been  linked  with 
improving  the  social  and  psychological  environments  of  healthcare  settings  and  the  critical 
interpersonal  relationships  between  physicians  and  other  healthcare  professionals  (82).  A 
systemic  review  from  Aarons  (1)  found  a  link  between  effective  leadership  and  improved 
evidence-based  practices  in  mental  health  settings.  Although  intended  for  faculty  to  evaluate 
student  physicians  at  USU,  the  MMLA  has  potential  to  be  used  among  numerous  healthcare 
organizations  and  various  professionals  to  improve  the  quality  of  care,  patient  safety,  and  patient 
satisfaction. 

Moreover,  the  MMLA  also  may  be  used  as  a  self-  or  peer  assessment  instrument.  The 
items  and  scale  do  not  necessarily  require  a  senior  educator  or  evaluator.  In  his  landmark  article 
describing  the  nature  of  the  therapeutic  alliance  in  psychotherapy,  Ackerman  (3)  asserted  that 
confidence,  effective  communication,  trustworthiness,  and  adaptability  were  all  predictive  of  a 
healthy  therapist-patient  alliance.  Each  of  these  items  are  directly  evaluated  by  the  MMLA. 
Ackerman  (3)  also  predicted  that  warmth  and  openness  as  essential  parts  of  the  therapeutic 
alliance,  which  may  arguably  be  measured  by  the  level  of  Self-awareness  and  emotional 
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intelligence  included  in  the  MMLA.  The  MMLA  may  be  a  valuable,  brief  self-assessment  for 
the  healthcare  provider  who  desires  to  gain  more  perspective  on  their  skills  in  building  and 
maintaining  a  therapeutic  alliance.  Although  the  image  of  a  physician  caring  for  a  patient  may 
not  conjure  up  images  of  GEN  Washington  crossing  the  icy  Delaware  River,  or  LTG  (R)  Russel 
Honore  turning  the  tide  for  Hurricane  Katrina-ravaged  New  Orleans,  the  physician’s  strong 
character,  effective  communication,  and  competence  in  multiple  environments  is  and  will  remain 
a  vital  component  of  leadership. 
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CHAPTER  6:  Conclusion 


This  project  used  a  standard  Delphi  technique,  a  structured  qualitative  communication 
method,  to  communicate  with  a  diverse  group  of  leadership  experts  in  order  to  address  three 
broad  questions  presented  in  Chapter  1 : 

1)  What  do  expert  and  successful  leaders  believe  is  important  in  assessing  leadership? 

2)  How  would  a  potential  leadership  assessment  instrument  be  created  from  expert 
consensus  and  guide  leadership  education  and  development  in  military  medicine? 

3)  Can  the  expert  consensus  be  incorporated  into  medical  education  and  leadership 
development? 

In  addressing  the  first  and  second  questions,  the  leadership  experts  from  the  initial  Delphi 
panel  of  experts  included  33  leadership  experts  who  represent  all  branches  of  the  uniformed 
services,  multiple  healthcare  professions,  and  civilian  community  leaders.  Analysis  of  the 
panel’s  responses  and  subsequent  refinement  from  the  project’s  coding  team  and  a  cognitive 
interview  of  15  healthcare  educators  created  an  assessment  instrument  of  24  items  and  criteria 
measured  by  a  behavior-based  frequency  scale.  The  resulting  instrument,  called  the  Military 
Medical  Leadership  Assessment  (MMLA),  is  a  brief,  3  page  assessment  that  may  be  used  by 
USU  faculty,  student  peers,  and  as  a  self-assessment.  This  project  was  the  first  step  in  the 
creation  of  the  MMLA,  which  requires  formal  validation  in  future  studies. 

Addressing  the  third  question  about  expert  consensus  integrated  within  medical  education 
and  leadership  development.  The  MMLA  is  especially  designed  to  be  integrated  within  the  USU 
Leadership  Education  and  Development  (LEAD)  and  develop  the  USU  FourCe  Model  of 
Medical  Leadership.  In  addition  to  providing  useful  feedback  on  student  physician  performance 
and  the  efficacy  of  the  leadership  development  program  at  USU,  the  instrument  should  also  be 
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used  to  provide  data  to  advance  and  improve  existing  models  of  medical  leadership,  as  well  as 
compare  leadership  competency  of  USU  medical  students  with  their  peers  at  civilian  institutions. 

The  MMLA  is  the  first  Joint-Service  assessment  that  is  face  and  content  valid 
specifically  designed  to  provide  feedback  for  leaders  within  the  Military  Healthcare  System 
(MHS).  In  the  future,  the  MMLA  should  be  formally  evaluated  for  criterion  and  construct 
validity  among  students  at  USU  and  leadership  development  programs  throughout  the  MHS. 
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APPENDIX  A:  Project  Timeline 


MMLAQ2  MMLACI1 


30  Days 
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APPENDIX  B:  Study  Surveys 


URSi  Uniformed  Services  university 

of  the  Health  Sciences 


MMLA  Q1 


Demographics 

Please  tell  us  a  little  about  yourself 

1.  Are  you  male  or  female? 

Female 
'  Male 

2.  What  is  your  age? 

C  j  17 or  younger 

13-20 
J  21-29 
Q  30-39 
4049 
0  50-59 

60  or  older 
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3.  What  is  the  highest  level  of  school  you  have  completed  or  the  highest  degree  you  have  received? 

High  school  degree  or  equivalent  (e.g.,  GED) 

Some  college  but  no  degree 
Associate  degree 
Bachelor  degree 
Master's  degree 

Doctor  of  Medicine/  Doctor  of  Osteopathy  (MD/DO) 

Doctor  of  Philosophy  (PhD) 

Other  Professional  Doctorate  (DNR  PsyD,  OD,  PharmD) 

Other  (please  specify) 


4.  Are  you  White,  Black  or  African-American,  American  Indian  or  Alaskan  Native,  Asian,  Native  Hawaiian  or 
other  Pacific  islander,  or  some  other  race? 

White 

Black  or  African-American 
American  Indian  or  Alaskan  Native 
Asian 

Native  Hawaiian  or  other  Pacific  Islander 
From  multiple  races 
Some  other  race  (please  specify) 


5.  Are  you  Mexican-American,  Chicano,  Puerto  Rican,  Cuban-American,  or  some  other  Spanish,  Hispanic, 
or  Latino  group? 

I  am  not  Spanish,  Hispanic,  or  Latino 

Mex  ican-America  n 

Chicano 

Puerto  Rican 

Cuban-American 

Some  other  Spanish,  Hispanic,  or  Latino  group 
From  multiple  Spanish,  Hispanic,  or  Latino  groups 

2 
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6.  Which  branch  of  uniformed  service  have  you  served  or  are  currently  serving? 


Army 

Marine  Corps 
Navy 

(  ;  Air  Force 
Coast  Guard 
Public  Health  Service 
N/A  (Civilian) 

7.  What  best  describes  your  medical  or  professional  specialty? 
(  )  Physician 
Dentist 
Nurse 

Healthcare  Admin  (Uniformed) 

Healthcare  Admin  (Civilian) 

Doctoral-level  non-physician  provider  (OD,  PT,  PhD/PsyD) 

Other  non-physician  provider  (PA,  RD) 

Military  Line  Commander  (Non-Medical) 

Non-Commissioned  Officer  (NCO)  -  Medical 
Non-Commissioned  Officer  (NCO)  -  Non-Medical 
Other  (please  specify) 


8.  Please  select  your  highest  uniformed  rank  achieved 
O  E1-E6 

O  E7‘E9 

01-03 
Q  04-05 

O  06 

Q  07-09 
C  ')  N/A  (Civilian) 


3 
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HIM  Uniformed  Services  University 

of  the  Health  Sciences 


MMLA  Q1 


Leadership 


Please  tell  us  about  your  leadership  experlence(s)  and  what  you  think 

9.  Briefly,  please  describe  your  leadership  experience(s)  to  the  present  day 


10.  In  the  box  below,  please  list  any  terms,  concepts,  or  ideas  you  believe  are  important  to  developing 
medical  leadership  skills  in  uniformed  medical  students 


Thank  you  for  your  responses.  Once  the  responses  from  other  experts  are  collected;  another  questionnaire  will  be  emailed  to  you  for 
further  input 


4 
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[IIW  Uniformed  Services  University 

^  v-yp  °f  l^e  health  Sciences 


Survey  2.  Rating  Importance 


Please  review  the  following  terms  that  have  been  generated  by  you  and  other  experts  and  rate  their 
importance  in  teaching  and  assessing  leadership  to  uniformed  medical  students.  Thank  you  for 
your  participation. 


1.  How  important  are  the  following  terms  &  phrases? 


Not  Important 

Slightly  Important 

Somewhat 

Important 

Moderately 

Important 

Very  Important 

Accountability 

O 

O 

O 

O 

O 

Knowing  the  Chain  of 

Command 

O 

o 

o 

O 

o 

Financial  Responsibility 

o 

o 

o 

o 

■m 

Risk  Management 

o 

o 

o 

o 

o 

Time  Management 

o 

o 

o 

o 

o 

Communication  Skills 

o 

o 

o 

o 

o 

Listening 

o 

o 

o 

o 

o 

Oral  Communication 

Skills 

o 

o 

o 

o 

o 

Actively  Participating 

o 

o 

o 

o 

mm 

Collaborating 

o 

o 

o 

o 

o 

Energizing 

o 

o 

o 

o 

o 

Empowering  Others 

o 

o 

o 

o 

o 

Delegating 

o 

o 

o 

o 

o 

Leading  by  Example 

o 

o 

o 

o 

o 

Motivating  Others 

o 

o 

o 

o 

o 
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2.  How  important  are  the  following  terms  &  phrases? 


Not  Important 

Slightly  Important 

Somewhat 

Important 

Moderately 

Important 

Very  Important 

Sponsorship 

O 

O 

O 

O 

O 

Serving  Others 

O 

o 

o 

o 

o 

Mentoring 

o 

o 

o 

o 

o 

Managing 

o 

o 

o 

o 

o 

Problem  Solving  Skills 

o 

o 

o 

o 

o 

Decision  Making 

o 

o 

o 

o 

o 

Prioritizing 

o 

o 

o 

o 

o 

Considering 

Perspectives 

o 

o 

o 

o 

o 

Thinking  Strategically 

o 

o 

o 

o 

o 

Balancing  Priorities 

o 

o 

o 

o 

o 

Emotional  Intelligence 

o 

o 

o 

o 

o 

Self  Awareness 

o 

o 

o 

o 

o 

Remain  Calm 

o 

o 

o 

o 

o 

Physical  Strength 

o 

o 

o 

o 

o 

Learning 

o 

o 

o 

o 

o 
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3.  How  important  are  the  following  terms  &  phrases? 


Not  Important 

Slightly  Important 

Somewhat 

Important 

Moderately 

Important 

Very  Important 

Conflict  Resolution 

O 

O 

O 

O 

O 

Operational  Astuteness 

O 

o 

O 

o 

o 

Cultural  Competency 

o 

o 

o 

o 

o 

Keeping  Skills  Current 

o 

o 

o 

o 

o 

Medical  Planning 

o 

o 

o 

o 

o 

Mission  Focus 

o 

o 

o 

o 

o 

Tactical  Combat  Care 

o 

o 

o 

o 

o 

Understanding  Teams 

o 

o 

o 

o 

o 

Understanding  Service 

H  istory 

o 

o 

o 

o 

o 

Understanding  Mental 

Trauma 

o 

o 

o 

o 

o 

Adaptability 

o 

o 

o 

o 

o 

Flexibility 

o 

o 

o 

o 

o 

Authenticity 

o 

o 

o 

o 

o 

Loyalty 

o 

o 

o 

o 

o 

Confidence 

o 

o 

o 

o 

o 

Organizational  Values 

o 

o 

o 

o 

o 
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4.  How  important  are  the  following  terms  &  phrases? 

Not  Important 

Slightly  Important 

Somewhat 

Important 

Moderately 

Important 

Very  Important 

Command  Presence 

O 

O 

O 

O 

O 

Creativity 

O 

o 

o 

o 

o 

Innovation 

o 

o 

o 

o 

o 

Integrity 

o 

o 

o 

o 

o 

Dedication 

o 

o 

o 

o 

o 

Empathy 

o 

o 

o 

o 

o 

Kindness 

o 

o 

o 

o 

o 

Patience 

o 

o 

o 

o 

o 

Self-Restraint 

o 

o 

o 

o 

o 

Showing  Vulnerability 

o 

o 

o 

o 

o 

Selflessness 

o 

o 

o 

o 

o 

Humility 

o 

o 

o 

o 

o 

Trustworthiness 

o 

o 

o 

o 

o 

Excellence 

o 

o 

o 

o 

o 

Personal  Values 

o 

o 

o 

o 

o 

Responsibility 

o 

o 

o 

o 

o 
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5.  How  important  are  the  following  terms  &  phrases? 


Not  Important 

Slightly  Important 

Somewhat 

Important 

Moderately 

Important 

Very  Important 

P  hy  si  ca  1  Att  ra  ct  i  ven  ess 

O 

O 

O 

O 

O 

Charisma 

O 

o 

O 

O 

o 

Popu  la  rity 

o 

o 

o 

o 

o 

Spirituality 

o 

o 

o 

o 

o 

Valorous 

0 

o 

o 

o 

o 

Religiousness 

o 

o 

o 

o 

o 

Hard  Working 

o 

o 

o 

o 

o 

Family  Values 

o 

o 

o 

o 

o 
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This  will  be  the  final  survey  you  will  be  asked  to  complete.  Below  are  30  items  that  you  and  other 
experts  have  identified  as  moderately  important  to  very  important.  Please  categorize  these  items 
into  one  of  four  major  categories:  Competence;  Communication;  Character;  Context.  Some  items 
may  fit  into  more  than  one  category;  please  choose  the  one  category  that  best  describes  the  item. 
Thank  you  for  your  valuable  input. 

1.  Which  Category  Best  Describes  the  Following  Terms? 


Competence 

Communication 

Character 

Context 

Accountability 

O 

O 

O 

O 

Problem  Solving 

O 

o 

o 

o 

Time  Management 

HUMAN 

o 

o 

o 

Emotional  Intelligence 

o 

o 

o 

o 

Collaborating 

o 

o 

o 

o 

Motivating 

o 

o 

o 

o 

Decision  Making 

o 

o 

o 

o 

Learning 

o 

o 

o 

o 

Mentoring 

o 

o 

o 

o 

Valor 

o 

o 

o 

o 
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2.  Which  Category  Best  Describes  the  Following  Terms? 


Competence 

Communication 

Character 

Context 

Oral  Communication 

O 

O 

O 

O 

Non-Verbal 

Communication 

O 

o 

o 

o 

Written  Communication 

o 

o 

o 

o 

Listening 

o 

o 

o 

o 

Leading  by  Example 

o 

o 

o 

o 

Selflessness 

o 

o 

o 

o 

Self-Awareness 

o 

o 

o 

o 

Patience 

o 

o 

o 

o 

Adaptability 

o 

o 

o 

o 

Integrity 

o 

o 

o 

o 
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3.  Which  Category  Best  Describes  the  Following  Terms? 


Competence 

Communication 

Character 

Context 

Responsibility 

O 

O 

O 

O 

Creativity 

O 

o 

o 

o 

Confidence 

o 

o 

o 

o 

Charismatic 

o 

o 

o 

o 

Conflict  Resolution 

o 

o 

o 

o 

Mission  Focus 

o 

o 

o 

o 

Adapting  to  different 

environments 

o 

o 

o 

o 

Cultural  Sensitivity 

o 

o 

o 

o 

Popu  la  rity 

o 

o 

o 

o 
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Welcome  Page 

Each  of  the  items  listed  in  this  survey  will  be  used  in  an  assessment  of  uniformed  medical 
leadership. 

There  are  25  items  each  on  a  separate  page.  Please  rate  each  item  for  clarity  and  offer  any 
suggestions  to  make  the  item  more  clear. 

Click  below  to  begin  the  survey. 

Thank  you. 


1 
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Item  1;  Please  assess  the  Question  Below 


"How  often  does  the  rated  student  demonstrate  problem  solving 
skills?" 


1.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

o 

o 

o 

o 

2.  Please  add  any  comments  that  you  feel  would  Improve  the  quality  of  the  question 


103 


■IM  Uniformed  Services  University 

of  the  Health  Sciences 


MMLA  CM 


Item  2:  Please  Assess  The  Question  Below 


"How  often  does  the  rated  student  practice  effective  time 
management?" 


3.  How  dear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

o 

o 

o 

o 

4.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  3.  Please  Assess  The  Question  Below 

"How  often  does  the  rated  student  demonstrate  effective  decision 
making?" 

5.  How  dear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

6.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  4.  Please  Assess  The  Question  Below 

"How  often  does  the  rated  student  continues  to  learn?" 

7.  How  dear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

8.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  5  Please  Assess  the  Question  Below 

"How  often  does  the  rated  student  remain  focused  on  the 
mission?" 

9.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

10.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  6:  Please  Assess  the  Question  Below 

"How  often  does  the  rated  student  demonstrates  effective  oral 
communication?" 

11.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

12.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  7  Please  Assess  the  Question  Below 

"How  often  does  the  rated  student  demonstrates  effective  non¬ 
verbal  communication?" 


13.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

14.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  8;  Please  Assess  the  Question  Below 

"How  often  does  the  rated  student  demonstrates  effective  written 
communication?" 

15.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

16.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  9:  Please  Asses  the  Question  Below 


"How  often  does  the  rated  student  demonstrates  effective  listening 
skills?" 


17.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

18.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  14:  Please  Assess  the  Question  Below 


"How  often  does  the  rated  student  demonstrate  a  high  level  of 
emotional  intelligence?" 


27.  How  dear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

28.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  24:  Please  Assess  the  Question  Below 

"How  often  does  the  rated  student  adapts  to  different 
environments?" 

47.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

48.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Item  25:  Please  Assess  the  Question  Below 


"How  often  does  the  rated  student  shows  cultural  sensitivity?" 

49.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

50.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  question 
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Welcome  Page 


Each  of  the  items  presented  in  this  survey  have  incorporated  your  valuable  feedback  from  the 
previous  survey. 

There  are  24  assessment  items  each  on  a  separate  page.  Please  rate  each  item  and  the  criteria 
presented  with  the  item  for  clarity  and  offer  any  suggestions  to  make  the  item  more  clear. 

Each  item  uses  the  following  behaviorally-based  scale  to  assess  the  rated  student: 

[Almost  Never— Infrequently— Sometimes— Often—  Almost  Always--Not  Observed] 

Click  below  to  begin  the  survey. 


Thank  you. 
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Item  1  Problem  Solving  Skills 

Problem  solving  skills  includes  skills  required  to  effectively  identify  a  problem  (clinical,  academic, 
or  military),  integrate  background  knowledge  of  the  problem,  brainstorm  possible  solutions  and 
hypotheses,  and  determine  what  additional  information  is  needed  to  solve  the  problem  and/or 
execute  the  hypotheses. 

"How  often  does  the  rated  student  demonstrate  problem  solving 
skills?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 

1.  How  dear  is  this  criteria? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

2.  How  clear  is  this  question? 

Not  at  a  1 1  S I  ig  ht  ly  Clea  r  Mod  erately  C  lea  r 

o  o  o 

3.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


2 


Very  Clear 

o 
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Item  2;  Time  Management 


Time  management  includes  skills  and  techniques  required  to  effectively  and  adequately  prepare 
and  successfully  execute  an  assignment,  task,  or  mission  given  specific  time  constraints. 

"How  often  does  the  rated  student  demonstrate  effective  time 
management?" 

[Almost  Never — Infrequently — Sometimes— Often — Almost  Always — Not  Observed] 


4.  How  clear  is  this  criteria? 


Slightly  Clear 


Moderately  Clear 


Very  Clear 


5.  How  clear  is  this  question? 


Slightly  Clear 


Moderately  Clear 


Very  Clear 


6.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  3:  Decision  Making 

Decision  making  includes  the  range  of  cognitive  processes  where  the  evaluated  student  identifies 
and  selects  the  best  course  of  action  among  more  than  one  alternative  given  the  constraints  of  the 
environment  and  time. 

"How  often  does  the  rated  student  demonstrate  effective  decision 
making?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


7.  How  dear  is  this  criteria? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

8.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

9.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


131 


Uniformed  Services  University 

of  the  Health  Sciences 


MMLACI2 


Item  4:  Learning 

Effective  learning  includes  the  continuous  cognition  (i.e.,  thoughts),  behaviors,  and  motivations 
necessary  for  the  evaluated  student  to  adapt  and  master  competencies  required  to  be  an 
outstanding  uniformed  physician. 

"How  often  does  the  rated  student  continue  to  learn?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


10.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

12.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  crtieria  and/or  question 
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Item  5:  Mission  Focus 

Mission  focus  includes  cognitive  processes,  behaviors,  and  competencies  required  for  the 
evaluated  student  to  remain  committed  to  successfully  achieving  a  given  mission  (i.e.,  a  mission 
from  a  higher  authority)  and  the  mission  of  the  organization. 

"How  often  does  the  rated  student  remain  focused  on 
given  mission(s)  and  the  mission  of  the  university?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 

13.  How  clear  is  this  criteria? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

14.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

15.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


6 
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Item  6:  Oral  Comunication 

Effective  oral  communication  includes  skills,  competencies,  and  behaviors  necessary  for  the 
evaluated  student  to  express  information  by  speaking  and  have  that  information  understood  by  the 
intended  recipient(s). 

"How  often  does  the  rated  student  demonstrate  effective  oral 
communication?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


16.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

17.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

18.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  7.  Non-verbal  Communication 

Non-verbal  communication  includes  behaviors  required  to  complement  verbal  communication  so 
that  the  information  conveyed  is  better  understood  by  the  intended  recipient(s).  Examples  include 
hand  gesture,  eye  contact,  body  posture,  and  facial  expressions. 

"How  often  does  the  rated  student  demonstrate  effective  non¬ 
verbal  communication?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


19.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

20.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

21.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


8 


135 


■I PN  Uniformed  Services  University 

of  the  Health  Sciences 


MMLACI2 


Item  8;  Written  Communication 

Written  communication  includes  any  skills,  traits,  and  competencies  required  to  effectively  convey 
information  through  written  words  or  symbols  and  that  the  information  is  understood  by  the 
intended  recipient(s). 

"How  often  does  the  rated  student  demonstrate  effective  written 
communication?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


22.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

23.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

24.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


9 
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Item  9:  Listening  Skills 

Listening  skills  includes  the  cognitive  processes  and  behaviors  required  to  effectively  hear  and 
understand  audible  communication. 

"How  often  does  the  rated  student  demonstrate  effective  listening 
skills?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 

25.  How  dear  is  this  criteria? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

26.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

27.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


10 
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Item  10;  Collaboration 

Collaboration  includes  skills  and  competencies  required  for  the  evaluated  student  to  effectively 
work  jointly  with  others  especially  given  physical,  psychological  (i.e.,  under  stress),  ortime 
constraints. 

"How  often  does  the  rated  student  collaborate  with  others?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


28.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

29.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

30.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  crtieria  and/or  question 


11 
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Item  11:  Motivation 

Motivating  others  includes  skills,  traits,  competencies,  and  behaviors  necessary  to  effectively 
influence  others  to  commit  and  achieve  individual  and  group  goals.  Motivating  others  should  not 
rely  on  averse  leadership  behaviors  (i.e.,  toxic  leadership)  but  should  focus  on  person-based 
interactions  designed  to  intrinsically  motivate  peers  and  subordinates. 

"How  often  does  the  rated  student  motivate  others?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


31.  How  clear  is  this  criteria? 

Not  at  all  Slightly  Clear 

o  o 


Moderately  Clear 

O 


Very  Clear 

O 


32.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear 

O  O 


Moderately  Clear 

O 


Very  Clear 

O 


33.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  12:  Mentorship 

Mentorship  includes  the  interpersonal  skills  and  relationships  by  which  the  rated  student  guides 
others  given  the  rated  student's  competency  or  expertise. 

"How  often  does  the  rated  student  mentor  others?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


34.  How  dear  is  this  criteria? 

Not  at  all  Slightly  Clear 

O  O 


Moderately  Clear 

O 


Very  Clear 

O 


35.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear 

O  O 


Moderately  Clear 

O 


Very  Clear 

O 


36.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  13:  Accountability 

Accountability  includes  those  behaviors  and  attitudes  required  for  the  rated  student  to  hold 
themselves  answerable  to  individual  and  organizational  answerability  and  liability. 

"How  often  does  the  rated  student  demonstrate  accountability?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


37.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

38.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

39.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


14 
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Item  14;  Emotional  Intelligence 

Emotional  Intelligence  Includes  those  skills,  traits,  and  behaviors  by  which  the  rated  student  Is 
aware  and  manages  their  own  emotions  and  is  aware  of  the  emotions  of  others. 

"How  often  does  the  rated  student  demonstrate  a  high  level  of 
emotional  intelligence?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 

40.  How  dear  is  this  criteria? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

41.  Hqw  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

42.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


15 
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Item  15:  Leading  by  Example 

Leading  by  example  Includes  how  well  the  rated  student  uses  their  own  leadership  behaviors, 
skills,  and  traits  to  provide  a  high  standard  for  others  to  emulate. 

"How  often  does  the  rated  student  lead  by  example?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


43.  How  dear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

44.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

45.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


16 
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Item  16;  Selflessness 

Selflessness  Includes  the  behaviors  and  traits  by  which  the  rated  student  places  the  needs  of 
others  (peers,  subordinates,  and  the  organization)  before  their  own. 

"How  often  does  the  rated  student  demonstrate  selflessness?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 

46.  How  dear  is  this  criteria? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

47.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

48.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


17 
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Item  17:  Self-Awareness 

Self-Awareness  is  the  introspective  ability  for  the  rated  student  to  recognize  their  own  actions  as 
deliberate  and  distinct  from  others.  In  military  medical  leadership,  Self-Awareness  also  includes 
the  capacity  for  the  rated  student  to  make  objective  evaluations  of  their  own  traits,  behaviors,  and 
beliefs. 

"How  often  does  the  rated  student  demonstrate  self-awareness?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


49.  How  dear  is  this  criteria? 

Not  at  all  Slightly  Clear 

o  o 


Moderately  Clear 

O 


Very  Clear 

O 


50.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear 

O  O 


Moderately  Clear 

O 


Very  Clear 

O 


51.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  18:  Patience 

Patience  includes  resiliency,  endurance,  and  capacity  to  persevere  under  difficult  or  challenging 
circumstances  without  resorting  to  negative  emotions  and  behaviors.  Patience  in  military  medical 
leadership  includes  interactions  with  peers,  patients,  subordinates,  and  leaders. 

"How  often  does  the  rated  student  display  patience?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


52.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

53.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

54.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  19:  Adaptability 

Adaptability  includes  behaviors,  traits,  and  cognitions  required  to  effectively  respond  to  changes  in 
the  physical,  psychological,  and  social  environment.  Additionally,  adaptability  is  also  a  measure  of 
the  rated  student  to  cope  with  unexpected  changes. 

"How  often  does  the  rated  student  demonstrate  adaptability?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


55.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

56.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

57.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  21:  Responsibility 


Responsibility  includes  the  behaviors  and  traits  necessary  for  the  rated  student  to  hold  themselves 
personally  accountable  to  ensure  personal  and  organizational  success,  as  well  as  the  motivation  to 
act  independently  when  appropriate. 

"How  often  does  the  rated  student  take  responsibility?" 

[Almost  Never — Infrequently — Sometimes— Often — Almost  Always — Not  Observed] 


61.  How  dear  is  this  criteria? 

Not  at  all  Slightly  Clear 

O  O 


Moderately  Clear 

O 


Very  Clear 

O 


62.  How  dear  is  this  question? 

Not  at  all  Slightly  Clear 

o  o 


Moderately  Clear 

O 


Very  Clear 

O 


63.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  22;  Confidence 

Confidence  includes  traits  and  behaviors  that  indicate  the  rated  student  has  self-assurance  in  their 
own  abilities  and  competence.  In  military  medical  leadership,  confidence  also  includes  an  attitude 
conducive  to  continued  personal  and  professional  growth. 

"How  often  does  the  rated  student  display  confidence?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


64.  How  clear  is  this  criteria? 


Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

65.  How  clear  is  this  question? 

Not  at  all 

Slightly  Clear 

Moderately  Clear 

Very  Clear 

O 

O 

O 

O 

66.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  23:  Creativity 


Creativity  includes  the  cognitive  capacity  for  the  rated  studentto  generate  new  ideas  and  solutions. 
Creativity  in  military  medical  leadership  also  includes  traits  and  behaviors  required  to  generate 
products  or  ideas  that  are  useful  and  (when  applicable)  improve  previous  models  and  paradigms. 


"How  often  does  the  rated  student  show  creativity?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 


67.  How  clear  is  this  criteria? 

Not  at  all  Slightly  Clear 

o  o 


Moderately  Clear 

O 


Very  Clear 

O 


68.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear 

O  O 


Moderately  Clear 

O 


Very  Clear 

O 


69.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 
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Item  24:  Cultural  Sensitivity 

Cultural  sensitivity  includes  awareness  of  how  differences  and  similarities  among  different  cultures 
influence  values,  learning,  and  behavior.  Cultural  sensitivity  in  military  medical  leadership  also 
includes  self-awareness  of  the  rated  student's  own  culture  and  how  that  influences  interactions 
with  peers,  subordinates,  leaders,  and  patients. 

"How  often  does  the  rated  student  show  cultural  sensitivity?" 

[Almost  Never — Infrequently— Sometimes— Often — Almost  Always — Not  Observed] 

70.  How  clear  is  this  criteria? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

71.  How  clear  is  this  question? 

Not  at  all  Slightly  Clear  Moderately  Clear  Very  Clear 

o  o  o  o 

72.  Please  add  any  comments  that  you  feel  would  improve  the  quality  of  the  criteria  and/or  question 


25 
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APPENDIX  C:  Study  Information  Sheet 


Information  Sheet  (Delphi  Panel) 


1.  INTRODUCTION  OF  THE  STUDY 

You  are  being  asked  to  be  in  a  research  study  entitled,  Creating  a  Military  Medical  Leadership 
Assessment  Instrument  Using  the  Delphi  Method  at  the  Uniformed  Services  University  of  the  Health 
Sciences  (USUHS),  Bethesda,  Maryland.  You  have  been  identified  by  leadership  development  faculty  at 
the  Uniformed  Services  University  as  someone  who  has  valuable  experience  in  leadership.  This 
information  sheet  provides  information  about  the  research  study.  Once  you  understand  the  study,  you  can 
decide  if  you  want  to  take  part  in  this  research  study.  Your  decision  to  take  part  is  voluntary.  This  means 
you  are  free  to  choose  if  you  want  to  take  part  in  this  study.  By  completing  and  submitting  the 
questionnaire,  you  have  consented  to  participate  in  this  study.  If  you  do  not  wish  to  be  contacted  again, 
please  notify  the  primary  investigator,  CPT  Matthew  Moosey  at  mmoosev@usuhs.edu  and  there  will  be 
no  future  contact. 

2.  PURPOSE  AND  PROCEDURES 

The  purpose  of  this  study  is  to  create  a  new  assessment  tool  to  assess  medical  leadership  in 
uniformed  medical  students  at  USUHS.  The  questions  asked  will  be  related  to  leadership,  medical 
leadership,  and  military  medical  leadership.  This  information  will  help  us  understand  more  about  how  to 
assess  leadership  in  military  medicine  and  will  provide  feedback  on  the  overall  effectiveness  of  leadership 
development  programs  at  USUHS.  We  will  use  this  information  in  the  future  to  better  design  leadership 
development  at  USUHS  and  advance  medical  leadership  throughout  the  Military  Health  System  (MHS). 

This  research  study  consists  of  a  series  of  approximately  5-7  emailed  questionnaires  over  the 
course  of  6  months  (approximately  one  questionnaire  a  month).  These  questionnaires  are  designed  to 
identify  important  components  of  military  medical  leadership  and  create  an  assessment  tool  based  on  your 
input.  You  will  be  collaboratively  working  with  other  leadership  experts,  although  you  will  remain 
anonymous  to  each  other.  Your  identity  will  only  be  known  to  the  Primary  Investigator,  CPT  Moosey, 
and  his  research  associates.  It  will  take  you  approximately  30  minutes  to  complete  each  questionnaire. 
The  questionnaires  will  be  provided  through  a  link  to  the  study’s  website, 

https://www.research.net/s/MilMedLeadershipQl.  When  filling  out  the  questionnaires,  you  may  skip  any 
questions  you  do  not  wish  to  answer.  With  each  questionnaire,  you  will  be  provided  with  descriptive 
statistics  about  how  other  leadership  experts  answered  previous  questions.  The  first  questionnaire  will  ask 
you  to  provide  demographic  data  about  yourself  (age,  race/ethnicity,  gender,  educational  level, 
occupational  specialty)  and  a  brief  description  of  your  leadership  experiences.  All  of  this  personal 
information  will  be  kept  secure  on  the  study’s  website  and  on  Department  of  Defense  (DoD)  secured 
computers  at  USUHS.  At  the  end  of  the  study,  you  will  be  provided  with  a  final  copy  of  the  assessment 
tool  for  you  to  use  in  your  own  organizations  as  you  see  appropriate. 

3.  POSSIBLE  BENEFITS  FROM  BEING  IN  THIS  STUDY 

There  will  be  no  direct  benefit  to  you  from  participating  in  this  research. 
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4.  COMPENSATION 


There  is  no  financial  compensation  for  your  participation  in  this  research. 

5.  ALTERNATIVE  PROCEDURES/TREATMENT 

The  only  alternative  to  participating  in  this  study  is,  not  participating. 

6.  POSSIBLE  RISKS  OR  DISCOMFORTS  FROM  BEING  IN  THIS  STUDY 
There  are  no  known  expected  risks  or  discomforts  from  being  in  this  study. 

7.  RIGHT  TO  WITHDRAW 

You  may  decide  to  stop  taking  part  in  the  study  at  any  time.  Your  relations  with  the  faculty,  staff, 
and  USUHS  will  not  be  changed  in  any  way  if  you  decide  to  end  your  participation  in  the  study. 

8.  RECOURSE  IN  THE  EVENT  OF  INJURY 

If  at  any  time  you  believe  you  have  suffered  an  injury  or  illness  as  a  result  of  participating  in  this 
research  project,  you  should  contact  the  Director  of  Human  Research  Protections  Programs  at  the 
Uniformed  Services  University  of  the  Health  Sciences,  Bethesda,  Maryland  20814-4799  at  (301)  295- 
9534.  This  office  can  review  the  matter  with  you,  can  provide  information  about  your  rights  as  a  subject, 
and  may  be  able  to  identify  resources  available  to  you.  If  you  believe  the  government  or  one  of  the 
government's  employees  (such  as  a  military  doctor)  has  injured  you,  a  claim  for  damages  (money)  against 
the  federal  government  (including  the  military)  may  be  filed  under  the  Federal  Torts  Claims  Act. 
Information  about  judicial  avenues  of  compensation  is  available  from  the  University's  General  Counsel  at 
(301)  295-3028. 

9.  PRIVACY  AND  CONFIDENTIALITY 

All  information  you  provide  as  part  of  this  study  will  be  confidential  and  will  be  protected  to  the 
fullest  extent  provided  by  law.  Your  responses  to  our  questionnaire  will  be  maintained  in  password- 
protected  archives  at  USUHS.  All  records  related  to  this  study  will  be  accessible  to  those  persons  directly 
involved  in  conducting  this  study  and  members  of  the  USUHS  Institutional  Review  Board  (IRB),  which 
provides  oversight  for  protection  of  human  research  volunteers.  In  addition,  the  IRB  at  USUHS  and  other 
federal  agencies  that  help  protect  people  who  are  involved  in  research  studies  may  need  to  see  the 
information  you  give  us.  Other  than  those  groups,  records  from  this  study  will  be  kept  private  to  the 
fullest  extent  of  the  law.  Scientific  reports  that  come  out  of  this  study  will  not  use  your  name  or  identify 
you  in  any  way.  If  you  are  a  military  member,  please  be  advised  that  under  Federal  Law,  a  military 
member's  confidentiality  cannot  be  strictly  guaranteed. 

10.  CONTACT  FOR  QUESTIONS  OR  PROBLEMS 

If  you  have  questions  about  this  research,  you  should  contact  CPT  Matthew  Moosey,  the  lead 
researcher  at  301-758-5703  or  at  mmoosey@usuhs.edu.  Even  in  the  evening  or  on  weekends,  you  can 
leave  a  message  at  that  number.  If  you  have  questions  about  your  rights  as  a  research  subject,  you  should 


154 


call  the  Director  of  Human  Research  Protections  Programs  at  USUHS  at  (301)  295-9534.  She  is  your 
representative  and  has  no  connection  to  the  researcher  conducting  this  study. 

**IF  YOU  HAVE  ANY  QUESTIONS  PLEASE  FEEL  FREE  TO  ASK  THEM** 

By  clicking  on  the  enclosed  questionnaire  link,  you  indicate  that  you  have  read  the  explanation  of 
this  study  on  this  form,  the  procedures  have  been  reviewed,  and  all  your  questions  have  been  answered. 
You  understand  the  nature  of  the  study  and  volunteer  to  participate  in  it.  You  attest  that  you  meet  the 
requirements  for  participation  in  this  study.  You  understand  that  the  study  is  designed  for  research 
purposes  and  not  to  be  of  direct  benefit  to  you. 
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Information  Sheet  (Cognitive  Interview) 


1.  INTRODUCTION  OF  THE  STUDY 

You  are  being  asked  to  be  in  a  research  study  entitled,  Creating  a  Military  Medical  Leadership 
Assessment  Instrument  Using  the  Delphi  Method  at  the  Uniformed  Services  University  of  the  Health 
Sciences  (USUHS),  Bethesda,  Maryland.  You  have  been  identified  by  leadership  development  faculty  at 
the  Uniformed  Services  University  as  someone  who  has  valuable  experience  in  leadership  education. 

This  information  sheet  provides  information  about  the  research  study.  Once  you  understand  the  study, 
you  can  decide  if  you  want  to  take  part  in  this  research  study.  Your  decision  to  take  part  is  voluntary.  This 
means  you  are  free  to  choose  if  you  want  to  take  part  in  this  study.  By  completing  and  submitting  the 
questionnaire,  you  have  consented  to  participate  in  this  study.  If  you  do  not  wish  to  be  contacted  again, 
please  notify  the  primary  investigator,  CPT  Matthew  Moosey  at  mmoosey@usuhs.edu  and  there  will  be 
no  future  contact. 

2.  PURPOSE  AND  PROCEDURES 

The  purpose  of  this  study  is  to  create  a  new  assessment  tool  to  assess  medical  leadership  in 
uniformed  medical  students  at  USUHS.  The  questions  asked  will  be  related  to  leadership,  medical 
leadership,  and  military  medical  leadership.  This  information  will  help  us  understand  more  about  how  to 
assess  leadership  in  military  medicine  and  will  provide  feedback  on  the  overall  effectiveness  of  leadership 
development  programs  at  USUHS.  We  will  use  this  information  in  the  future  to  better  design  leadership 
development  at  USUHS  and  advance  medical  leadership  throughout  the  Military  Health  System  (MHS). 

This  research  study  consists  of  a  series  of  approximately  2-3  emailed  questionnaires  over  the 
course  of  2  months  (approximately  one  questionnaire  a  month).  These  questionnaires  are  designed  to 
clarify  create  an  assessment  tool  based  on  your  input.  You  will  be  collaboratively  working  with  other 
leadership  experts,  although  you  will  remain  anonymous  to  each  other.  Your  identity  will  only  be  known 
to  the  Primary  Investigator,  CPT  Moosey,  and  his  research  associates.  It  will  take  you  approximately  30 
minutes  to  complete  each  questionnaire.  The  questionnaires  will  be  provided  through  a  link  to  the  study’s 
website,  https://www.research.net/s/MilMedLeadershipCl.  When  filling  out  the  questionnaires,  you  may 
skip  any  questions  you  do  not  wish  to  answer.  At  the  end  of  the  study,  you  will  be  provided  with  a  final 
copy  of  the  assessment  tool  for  you  to  use  in  your  own  organizations  as  you  see  appropriate. 

3.  POSSIBLE  BENEFITS  FROM  BEING  IN  THIS  STUDY 

There  will  be  no  direct  benefit  to  you  from  participating  in  this  research. 

4.  COMPENSATION 

There  is  no  financial  compensation  for  your  participation  in  this  research. 

5.  ALTERNATIVE  PROCEDURES/TREATMENT 

The  only  alternative  to  participating  in  this  study  is,  not  participating. 

6.  POSSIBLE  RISKS  OR  DISCOMFORTS  FROM  BEING  IN  THIS  STUDY 
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There  are  no  known  expected  risks  or  discomforts  from  being  in  this  study. 

7.  RIGHT  TO  WITHDRAW 

You  may  decide  to  stop  taking  part  in  the  study  at  any  time.  Your  relations  with  the  faculty,  staff, 
and  USUHS  will  not  be  changed  in  any  way  if  you  decide  to  end  your  participation  in  the  study. 

8.  RECOURSE  IN  THE  EVENT  OF  INJURY 

If  at  any  time  you  believe  you  have  suffered  an  injury  or  illness  as  a  result  of  participating  in  this 
research  project,  you  should  contact  the  Director  of  Human  Research  Protections  Programs  at  the 
Uniformed  Services  University  of  the  Health  Sciences,  Bethesda,  Maryland  20814-4799  at  (301)  295- 
9534.  This  office  can  review  the  matter  with  you,  can  provide  information  about  your  rights  as  a  subject, 
and  may  be  able  to  identify  resources  available  to  you.  If  you  believe  the  government  or  one  of  the 
government's  employees  (such  as  a  military  doctor)  has  injured  you,  a  claim  for  damages  (money)  against 
the  federal  government  (including  the  military)  may  be  filed  under  the  Federal  Torts  Claims  Act. 
Information  about  judicial  avenues  of  compensation  is  available  from  the  University's  General  Counsel  at 
(301)  295-3028. 

9.  PRIVACY  AND  CONFIDENTIALITY 

All  information  you  provide  as  part  of  this  study  will  be  confidential  and  will  be  protected  to  the 
fullest  extent  provided  by  law.  Your  responses  to  our  questionnaire  will  be  maintained  in  password- 
protected  archives  at  USUHS.  All  records  related  to  this  study  will  be  accessible  to  those  persons  directly 
involved  in  conducting  this  study  and  members  of  the  USUHS  Institutional  Review  Board  (IRB),  which 
provides  oversight  for  protection  of  human  research  volunteers.  In  addition,  the  IRB  at  USUHS  and  other 
federal  agencies  that  help  protect  people  who  are  involved  in  research  studies  may  need  to  see  the 
information  you  give  us.  Other  than  those  groups,  records  from  this  study  will  be  kept  private  to  the 
fullest  extent  of  the  law.  Scientific  reports  that  come  out  of  this  study  will  not  use  your  name  or  identify 
you  in  any  way.  If  you  are  a  military  member,  please  be  advised  that  under  Federal  Law,  a  military 
member's  confidentiality  cannot  be  strictly  guaranteed. 

10.  CONTACT  FOR  QUESTIONS  OR  PROBLEMS 

If  you  have  questions  about  this  research,  you  should  contact  CPT  Matthew  Moosey,  the  lead 
researcher  at  301-758-5703  or  at  mmoosey@usuhs.edu.  Even  in  the  evening  or  on  weekends,  you  can 
leave  a  message  at  that  number.  If  you  have  questions  about  your  rights  as  a  research  subject,  you  should 
call  the  Director  of  Human  Research  Protections  Programs  at  USUHS  at  (301)  295-9534.  She  is  your 
representative  and  has  no  connection  to  the  researcher  conducting  this  study. 

**IF  YOU  HAVE  ANY  QUESTIONS  PLEASE  FEEL  FREE  TO  ASK  THEM** 

By  clicking  on  the  enclosed  questionnaire  link,  you  indicate  that  you  have  read  the  explanation  of 
this  study  on  this  form,  the  procedures  have  been  reviewed,  and  all  your  questions  have  been  answered. 
You  understand  the  nature  of  the  study  and  volunteer  to  participate  in  it.  You  attest  that  you  meet  the 
requirements  for  participation  in  this  study.  You  understand  that  the  study  is  designed  for  research 
purposes  and  not  to  be  of  direct  benefit  to  you. 
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APPENDIX  D:  IRB  Approval 


UNIFORMED  SERVICES  UNIVERSITY  OF  THE  HEALTH  SCIENCES 

4301  JOMEfi  BRIDGE  RQAD 
BET  ME  SEA,  IWmAMG  *)&14-4  7t2 
wwirtiA* 


February  12,2015 

MEMORANDUM  FOR  C?T  MA 1TKEW  MOOSEY.  S'lUDENl ,  DEPARTMENT  OF 
meric  a  i.  and  r:i  intcat.  psychology 

SI  J  BJ  IK  It':  LS1JMS  IRU  M  HAVA  0000 162H;  RoO  Aswunmcc  IWW  ;  Approval  ol'TO-72- 
3320  Tor  Human  Subjects  Participation 

CougiutuUiluins!  The  Initial  Review  for  your  No  More  Thau  Minima:  Risk  human 
subjects  research  protocol  TO-72-3326, entitled  "Creuling  a  Military  Medical  Leadership 
Assessment  Instrument  Using  tine  Delphi  Method."  was  reviewed  arul  approved  idr  execution  on 
February  12, 201 S  by  Ldimind  llowe.  M.D.,  J.D.  under  llie  provision  of  32  CU  R 
219.1lO(bXli'Suppl.  Fi7).  Ibis  apple  vat  will  lie  ie  pm  ted  to  the  IJ.SU  IRK  o'l  scheduled  in  meet 
on  March  12, 2015. 

The  pmpo.se  of  ib is  pref  ect  is  io  crest?  a  Military  Medical  Leadership  Assessment 
Instrument  fMMLA)  I’orUSU  faculty  in  assess  uni  lurmed  medical  students. 

Authorization  to  conduct  protocol  1 0-72 -3 32 h  wil.  automatically  term  in  ale  nr:  February 
1 1, 2016.  If  you  plan  to  conluuc  data  collection  oi  analysis  beyond  rhis  date.  IRB  approval  tor 
continuation  is  required.  F lease  submit  a  USU  Form  3204  A'K,  application  for  continuing 
approval  hft  days  prior  to  your  icrminalkm  dulc.  Yon  will  receive  a  reminder  Irons  JRBNel. 

You  are  required  to  submit  amendments  to  this  protocol,  changes  to  the  informed  consent 
document  (if  applicable),  adverse  event  reports,  andothei  information  pertinent  to  human 
research  for  this  project  in  JRBNct.  No  changes  to  this  protocol  may  be  implemented  prior  to 
TRR  approval.  If  you  have  questions  regarding  Ibis  JRR  action  or  questions  of  a  more  general 
naljure  concerning  human  purlieipu;ii>n  in  research.  please  contact  Trish  Heal}'  at  30 1  -295-3388 
or  pulrioi  ichealy  (if;  iisuhs.edu. 


Edmund  G.  Howe,  .VI. D..  J  .D. 
Chair  .  IKK  ft  1 


1Ik4i'U|IM  lihi  h.sNi  viPjli-1  .ai't^lfa'i.  AlK 

rjeclAai;‘ Siitnir.ure  Mullet:  In  axuroiiioe  wilk-LT* 1  inivtCTinvenL ^jptfovrk  Clini-iitor  acI'  iGPrA/ifc?  I..  ciidiiifd  ;/44  EW: 

iJiMj1  -«1e”.il  nnd  IX ID  np,>:i:.ihle  in'min-mru.  ri'recrra  nnl  rcpnhnirn--.  rroimnr-;  h&vie  Ik'cn  a  r  r.lm*  i.ttllv  $ -viwti  nn.1  ^;rr m  121*2  .nail  v.ho 
I «H*r  !x.l  1 lif-u-  |l«c  4J|iv  yl‘ iVl  *>  llK'ii  WvU  $J.~  ':%l  VH.it  VlT.  Ixniiui  lb .  I'.MIIKC  X'iSLill  U.ff  prcl'vu.'J 

L'ltclnnik  toy  iB  mi  J  jjsii*.  .ni*.' 


/  Concur^  IhonCOIlcury. 

'■  X^l  cici<Qz' 

Arthur  L,  Kclicrmailn.  MD,  Vll’ll 
Professor  tod  Dean.  School  of  Medicine 
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APPENDIX  E:  Final  Draft  of  MMLA 


Military  Medical  Leadership  Assessment  (MMLA)  Version  1.0 

Instructions  and  Item  Criteria 

Directions:  The  following  assessment  is  intended  to  assess  medical  leadership  skills  and 
proficiencies.  For  the  evaluated  student,  read  each  item  criteria  and  select  a  value  on  the 
attached  grade  sheet  based  on  your  observation  of  the  student  during  the  evaluated  period.  If  an 
item  was  not  observed  then  mark  “not  observed.” 

COMMUNICATION 

1.  Oral  communication  includes  skills,  competencies,  and  behaviors  to  express  information 
by  speaking  and  have  that  information  understood  by  the  intended  recipient(s). 

How  often  does  the  student  demonstrate  effective  oral  communication? 

2.  Non-verbal  communication  includes  behaviors  to  complement  verbal  communication  so  the 
information  conveyed  is  better  understood  by  the  intended  recipient(s)  (e.g.,  hand  gesture, 
eye  contact,  body  posture). 

How  often  does  the  student  demonstrate  effective  non-verbal  communication? 

3.  Written  Communication  includes  skills  and  competencies  to  effectively  convey  information 
through  written  words  or  symbols  and  that  the  information  is  understood  by  the  intended 
recipient(s). 

How  often  does  the  student  demonstrate  effective  written  communication? 

4.  Listening  effectively  includes  attention,  information  processing,  and  behaviors  to  understand 
audible  communication. 

How  often  does  the  student  demonstrate  effective  listening? 

5.  Collaborating  includes  attitudes  and  behaviors  to  effectively  work  with  other. 

How  often  does  the  student  demonstrate  collaboration? 

6.  Motivating  others  involves  influencing  others  to  care  about  and  work  to  achieve  individual 
and  group  goals. 

How  often  does  the  student  demonstrate  motivation  of  others? 

7.  Mentoring  includes  guiding  and  advising  others. 

How  often  does  the  student  demonstrate  mentoring  of  others? 

COMPETENCE 

8.  Problem  solving  includes  skills  to  effectively  identify  a  problem  (clinical,  academic,  or 
military),  integrate  background  knowledge  of  the  problem,  brainstorm  possible  solutions  and 
hypotheses,  and  determine  what  additional  information  is  needed  to  solve  the  problem. 

How  often  does  the  student  demonstrate  effective  problem  solving? 

9.  Decision  making  includes  thought  processes  identifying  and  selecting  the  best  course  of 
action  among  more  than  one  alternative. 

How  often  does  the  student  demonstrate  effective  decision  making? 


159 


10.  Time  management  involves  effectively  allocating  time  to  successfully  execute  a  task  or 
mission. 

How  often  does  the  student  demonstrate  effective  time  management? 

11.  Learning  effectively  involves  attitudes  and  behaviors  to  gain  new  knowledge  and  skills  or  to 
build  upon  past  knowledge  and  skills. 

How  often  does  the  student  demonstrate  learning? 

12.  Mission  focus  includes  attitudes  and  behaviors  required  to  successfully  achieving  a  given 
mission. 

How  often  does  the  student  demonstrate  mission  focus? 

CHARACTER 

13.  Self-awareness  is  the  ability  to  make  objective  evaluations  of  oneself. 

How  often  does  the  student  demonstrate  self-awareness? 

14.  Emotional  intelligence  involves  Self-awareness  and  Self-regulation  of  one’s  own  emotions 
and  understanding  with  empathy  the  emotions  of  others. 

How  often  does  the  student  demonstrate  emotional  intelligence? 

15.  Selflessness  includes  attitudes  (e.g.,  humility)  and  behaviors  that  places  needs  of  others 
(peers,  subordinates,  and  the  organization)  before  their  ones’  own. 

How  often  does  the  student  demonstrate  selflessness? 

16.  Leading  by  Example  includes  behaviors  that  provide  a  high  standard  for  others  to  emulate. 

How  often  does  the  student  demonstrate  leading  by  example? 

17.  Accountability  includes  those  attitudes  and  behaviors  to  hold  themselves  answerable  to 
individual  and  organizational  standards  necessary  to  be  responsible  for  one’s  actions. 

How  often  does  the  student  demonstrate  accountability? 

18.  Integrity:  Integrity  includes  traits,  behaviors,  and  beliefs  that  are  concurrent  with  strong  and 
internally  consistent  moral  principles. 

How  often  does  the  student  demonstrate  integrity? 

19.  Patience  includes  calm  demeanor,  resiliency,  endurance,  and  capacity  to  persevere  under 
difficult  or  challenging  circumstances  without  resorting  to  negative  emotions  and  behaviors. 

How  often  does  the  student  demonstrate  patience? 

20.  Adaptability  involves  responding  effectively  to  changes  in  the  physical,  psychological,  and 
social  environment. 

How  often  does  the  student  demonstrate  adaptability? 

21.  Confidence  includes  attitudes  and  behaviors  that  indicate  Self-assurance  in  one’s  own 
abilities  and  competence. 

How  often  does  the  student  demonstrate  confidence? 

22.  Responsibility  includes  attitudes  and  behaviors  to  be  accountable  to  ensure  personal  and 
organizational  success,  as  well  as  the  motivation  to  act  independently  when  appropriate. 

How  often  does  the  student  demonstrate  responsibility? 
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CONTEXT 


23.  Creativity  includes  generation  of  new  ideas  and  solutions. 

How  often  does  the  student  demonstrate  creativity? 

24.  Cultural  Sensitivity  includes  attitudes  and  behaviors  of  how  differences  and  similarities 
among  different  cultures  influence  values,  learning,  and  behavior.  Cultural  sensitivity  also 
includes  Self-awareness  of  one’s  own  culture  and  how  that  influences  interactions  with  peers, 
subordinates,  leaders,  and  patients. 

How  often  does  the  student  demonstrate  cultural  sensitivity? 
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Medical  Military  Leadership  Assessment  (MMLA)  Version  1.1  Scoresheet 

(Refer  to  the  Instructions  for  Evaluation  Criteria) 

Student  Name: 

Evaluator: 

Date: 

Item 

Almost  Never 
(0-20%  of  time) 

Infrequently 
(21-40%  of  time) 

Sometimes 
(41-60%  of  time) 

Often 

(61-80%  of  time) 

Almost  Always 
(81-100%  of  time) 

Not  Observed 

Comments 

COMMUNICATION 

Oral  Communication 

Non-Verbal 

Communication 

Written  Communication 

Listening 

Collaborating 

Motivating 

Mentoring 

COMPETENCE 


Problem  Solving 

Decision  Making 

Time  Management 

Learning 

Mission  F  ocus 

CHARACTER 


Self-Awareness 

Emotional  Intelligence 

Selflessness 

Leading  By  Example 

Accountability 

Integrity 

Patience 

Adaptability 

Confidence 

Responsibility 

CONTEXT 


Creativity 

Cultural  Sensitivity 

Overall  Comments: 
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APPENDIX  F :  Delphi  Panel  Demographics 


Gender 

N 

Professional  Specialty 

N 

Male 

18 

Physician 

6 

Female 

15 

Nurse 

5 

Age 

Dentist 

2 

21-29 

2 

Health  Care  Admin  (Mil) 

4 

30-39 

6 

Health  Care  Admin  (Civ) 

2 

40-49 

14 

Other  Non-Physician  Provider 

2 

50-59 

8 

Military  Line  Commander 

4 

60  or  older 

3 

NCO  -  Medical 

2 

Race  &  Ethnicity 

NCO  -  Line 

1 

Asian/  Pacific  Islander 

2 

Civilian  Community  Leader 

5 

Black/  African  American 

4 

Military  Rank 

Latino-a  /  Hispanic 

2 

E7-E9 

3 

Multiple  Races 

3 

01-03 

1 

White 

22 

04-05 

6 

Education  Level 

06 

14 

Some  College/  Associate’s 

4 

07-09 

2 

Bachelor’s 

3 

N/A  (Civilian) 

7 

Master’s 

13 

Percentages 

% 

Doctorate  (PhD,  PsyD,  DNP,  JD) 

4 

Ethnic  Minority 

33% 

MD/DO 

7 

Women 

45% 

DDS/DMD 

2 

Civilian 

20% 

Branch  of  Service 

Physicians 

18% 

Army 

8 

Nurses 

15% 

Marine  Corps 

2 

Military  Line  Leaders 

15% 

Navy 

8 

Civilian  Community  Leaders 

15% 

Coast  Guard 

1 

Civilian  Healthcare  Leaders 

9% 

Public  Health  Service 

2 

Enlisted 

10% 

Air  Force 

4 

Officers 

70% 

N/A  (Civilian) 

8 

Total  N 

33 
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APPENDIX  G:  Coding  Strategy 


M.J.  Moosey  April  14,  2015 

Introduction:  The  following  standard  operating  procedure  (SOP)  is  for  coders  assigned  to  the 
Military  Medical  Leadership  Assessment  (MMLA)  project.  This  SOP  is  meant  to  guide  the 
coder  through  the  process  of  taking  raw,  unedited  responses  by  the  Delphi  expert  panel  members 
and  categorizing  these  responses  in  a  way  that  helps  develop  a  tool  that  will  be  used  by  faculty  to 
assess  medical  students. 

BLUF :  As  a  coder,  you  will  read  the  open  ended  responses  of  the  Delphi  experts  and  interpret 
these  responses  in  such  a  way  that  will  support  the  creation  of  a  brief  (half  standard  8x10  page  to 
single  standard  8x10  page)  assessment  instrument. 

Current  form  of  the  Data:  The  first  questionnaire  (MMLA  Ql)  asked  the  expert  panel 
members  to  describe  their  personal  demographics  and  leadership  experience.  The  final  question 
asked  the  following: 

“please  list  any  terms,  concepts,  or  ideas  you  believe  are  important  to 
developing  medical  leadership  skills  in  uniformed  medical  students” 

Some  experts  wrote  detailed  narratives,  others  listed  bulleted  items,  while  others  wrote  a  mixture 
of  forms. 

How  the  Data  Need  to  be  Transformed:  Concepts  and  ideas  expressed  in  such  a  manner  that 
an  MEM  faculty  would  be  able  to  assess  an  individual  medical  student. 

The  Challenge:  There  is  no  “incorrect”  way  to  code  the  responses.  As  a  coder,  you  should  rely 
on  your  own  judgment  to  interpret  the  expert  panel’s  open  ended  responses  and  transform  these 
responses  in  a  way  that  will  be  used  to  create  an  assessment  tool. 

An  Example:  In  order  to  avoid  biasing  you  as  a  coder,  the  following  example  is  fictional  and 
NOT  taken  from  MMLA  Ql. 

Hypothetical  Question  1 :  Please  list  any  terms,  concepts,  or  ideas  you  believe  are  important  to 
developing  competency  in  WMTA  Metro  rail  operators. 

Hypothetical  Expert  1  Response:  A  Metro  Rail  operator  must,  above  all  else,  place  the  safety  of 
his  crew  and  passengers  as  paramount.  I  have  seen  far  too  many  preventable  accidents  due  to 
lackadaisical  safety  protocols.  Additionally,  a  Metro  rail  operator  should  be  a  clear 
communicator  and  should  train  extensively  on  new  communications  protocols.  Finally,  a  Metro 
rail  operator  should  be  flexible  and  prepared  to  educate  themselves  on  new  routes,  equipment, 
schedules,  and  customer  service. 
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Hypothetical  Expert  2  Response: 

Safety  and  training 

Commitment  and  dedication 

Customer  service 

Kindness 

Security 

Trust 

Comment  on  First  Response:  The  first  response  is  a  narrative  where  the  second  response  is  a  list. 
For  the  first  response,  you  will  need  to  read  each  sentence  then  list  words  that  will  accurately 
summarize  the  sentence  and  may  be  expressed  in  an  assessment  instrument.  For  instance, 

“A  Metro  Rail  operator  must,  above  all  else,  place  the  safety  of  his  crew  and  passengers  as 
paramount” 

Scan  for  the  most  essential  word  or  words  that  capture  the  essence  of  the  statement  using  as  few 
words  as  possible. 


CODE1:  Safety 

Potential  assessment  question  that  might  be  developed  based  on  Code  1:  How  well  does  the  rail 
operator  follow  WMTA  safety  protocols  for  their  crew  and  passengers? 

Note:  Do  not  worry  about  the  potential  assessment  question  because  there  may  be  numerous 
ways  to  ask  this  question.  The  experts  and  the  cognitive  interview  participants  will  address  those 
finer  stylistic  questions.  You  then  move  on  and  perform  the  same  analysis  and  coding  on  each 
sentence  in  the  narrative. 

Comment  on  Second  Response:  At  first  look,  the  second  response  might  appear  to  be  easier  to 
code  for  purposes  of  assessment  because  the  responses  are  already  expressed  as  singular 
concepts.  However,  be  careful  because  some  of  the  terms  should  be  separated  whereas  others 
may  be  coded  as  a  single  item.  For  instance 

“Safety  and  training” 


Safety  and  training,  although  related  in  some  instances,  might  be  different.  Therefore: 


165 


C0DE1:  Safety 
CODE2:  Training 


Potential  Assessment  Question  1:  How  well  does  the  rail  operator  follow  WMTA  safety 
protocols  for  their  crew  and  passengers? 


Potential  Assessment  Question  2:  How  well  does  the  rail  operator  leam  required  annual  rail 
operator  training? 


Conversely,  items  may  be  similar  enough  to  be  coded  as  a  single  item 


“Commitment  and  dedication” 


CODE3:  Commitment 


Potential  Assessment  Question  3:  How  well  does  the  rail  operator  commit  to  the  WMTA 
organizational  values? 


Conclusion:  Apply  sufficient  brain  power  to  appropriately  code  an  item  and  move  on.  Do  not 
overthink  your  coding  strategy  or  agonize  over  an  item.  Rely  on  and  trust  your  judgment. 
Additionally,  it  is  important  NOT  to  communicate  with  other  coders  about  your  work.  We  want 
to  minimize  potential  biases  in  your  coding.  Thank  you  again  for  your  time  and  efforts. 
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